adblock: release 1.2.0

* rework download/backup/restore handling
* rework firewall handling
* refine whitelist parsing
* changed default null-ip to 198.18.0.1 to prevent conflicts with bcp38
* spread traffic from ad servers (port 80/443) on two uhttpd instances,
this fixes timeouts on sites with many ssl ad server connections, i.e.
youtube.com
* provide statistics & last runtime for LuCI interface
* documentation update

Signed-off-by: Dirk Brenken <dev@brenken.org>

net/adblock/Makefile

@@ -7,7 +7,7 @@
 include $(TOPDIR)/rules.mk
 
 PKG_NAME:=adblock
-PKG_VERSION:=1.1.17
+PKG_VERSION:=1.2.0
 PKG_RELEASE:=1
 PKG_LICENSE:=GPL-3.0+
 PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org>

net/adblock/files/README.md

@@ -46,7 +46,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
     * [zeus tracker](https://zeustracker.abuse.ch)
     * => daily updates, approx. 440 entries
 * zero-conf like automatic installation & setup, usually no manual changes needed (i.e. ip address, network devices etc.)
-* supports a wide range of router modes (incl. AP mode), as long as the firewall and the DNS server are enabled
+* supports a wide range of router modes (incl. AP mode), as long as the firewall and the DNS server are enabled & in use
 * full IPv4 and IPv6 support
 * each blocklist source will be updated and processed separately
 * timestamp check to download and process only updated adblock list sources
@@ -54,14 +54,13 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * adblock source list parsing by fast & flexible regex rulesets
 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
 * quality checks during & after update of adblock lists to ensure a reliable dnsmasq service
-* basic adblock statistics via iptables packet counters
-* list states, (overall) list counts & last update time will be stored in uci config
+* adblock statistics, last runtime and list states/counts/update times will be stored in uci config for LuCI frontend
 * status & error logging to stdout and syslog
-* use a dynamic uhttpd instance as an adblock pixel server
-* use dynamic iptables rulesets for adblock related redirects/rejects
+* use two dynamic uhttpd instances as adblock pixel server, separated for ads delivered on port 80 and on port 443
+* use dynamic iptables chains/rulesets for adblock related redirects/rejects
 * init system support (start/stop/restart/reload/toggle)
 * hotplug support, the adblock start will be triggered by wan 'ifup' event
-* adblock toggle to quickly (temporary) switch adblocking 'on' or 'off'
+* adblock toggle to quickly switch adblocking 'on' or 'off'
 * optional: automatic adblock list backup/restore, backups will be (de-)compressed on the fly (disabled by default)
 * optional: add new adblock sources via uci config (see example below)
 
@@ -79,7 +78,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * adblock starts automatically during boot, triggered by wan-ifup event, check _logread -e "adblock"_ for adblock related information
 * optional: start/restart/stop the adblock service manually with _/etc/init.d/adblock_
 * optional: enable/disable your required adblock list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
-* optional: maintain the adblock service in luci under 'System => Startup'
+* optional: maintain the adblock service in LuCI under 'System => Startup'
 
 ## LuCI adblock companion package
 * for easy management of the various blocklist sources and adblock options there is also a nice & efficient LuCI frontend available
@@ -102,7 +101,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * **list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
 * **new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
 * **AP mode:** in AP mode adblock uses automatically the local router ip as nullip address. To make sure that your LuCI interface will be still accessible, please change the local uhttpd instance to ports <> 80/443 (see example below)
-* **Restricted mode:** to disable flash writes with adblock status information to the adblock config file (used by LuCI frontend), please set 'adb\_restricted' to '1'
+* **restricted mode:** to disable flash writes with adblock status information to the adblock config file (used by LuCI frontend), please set 'adb\_restricted' to '1'
 * **adblock toggle:** to quickly switch adblocking 'on' or 'off', simply use _/etc/init.d/adblock toggle_
 * **configuration update:** to update an outdated adblock config file with the current default version, please run _/etc/init.d/adblock cfgup_, make your individual changes and start the adblock service again
 * **debugging:** for script debugging please set the 'adb\_debug' variable in the header of _/etc/init.d/adblock_ to '1'
@@ -113,9 +112,10 @@ A lot of people already use adblocker plugins within their desktop browsers, but
     * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
     * adb\_cfgver => config version string (do not change!) - adblock will check this entry during startup
     * adb\_lanif => name of the logical lan interface (default: 'lan')
-    * adb\_nullport => port of the adblock uhttpd instance (default: '65535')
-    * adb\_nullipv4 => IPv4 blackhole ip address (default: '192.0.2.1', in AP mode: local router ip)
-    * adb\_nullipv6 => IPv6 blackhole ip address (default: '::ffff:c000:0201', in AP mode: local router ip)
+    * adb\_nullport => port of the adblock uhttpd instance used for ads delivered on port 80 (default: '65534')
+    * adb\_nullportssl => port of the adblock uhttpd instance used for ads delivered on port 443 (default: '65535')
+    * adb\_nullipv4 => IPv4 blackhole ip address (default: '198.18.0.1', in AP mode: local router ip)
+    * adb\_nullipv6 => IPv6 blackhole ip address (default: '::ffff:c612:0001', in AP mode: local router ip)
     * adb\_forcedns => redirect all local DNS queries to the local dnsmasq resolver (default: '1', enabled)
     * adb\_fetchttl => set the timeout for list downloads (default: '5' seconds)
     * adb\_restricted => disable updates of the adblock config file (no flash writes) during runtime (default: '0', disabled)
@@ -134,26 +134,27 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 <pre><code>
 ads.example.com
 
-This rule blocks:
+This entry blocks the following (sub)domains:
   http://ads.example.com/foo.gif
   http://server1.ads.example.com/foo.gif
   https://ads.example.com:8000/
 
-This rule doesn't block:
+This entry does not block:
   http://ads.example.com.ua/foo.gif
   http://example.com/
 </code></pre>
   
 **example whitelist entry (/etc/adblock/adblock.whitelist):**
 <pre><code>
-analytics.com
-
-This rule removes _all_ domains from the blocklists with this string in it, i.e.:
-  google-analytics.com
-  ssl.google-analytics.com
-  api.gameanalytics.com
-  photos.daily-deals.analoganalytics.com
-  adblockanalytics.com
+here.com
+
+This entry removes the following (sub)domains from the blocklists:
+  maps.here.com
+  here.com
+
+This entry does not remove:
+  where.com
+  www.adwhere.com
 </code></pre>
   
 **example uhttpd configuration in AP mode:**
@@ -171,20 +172,20 @@ config uhttpd 'main'
 grep "google-analytics.com" "/tmp/dnsmasq.d/adb_list"*
 
 This will output all matches with corresponding source files:
-  /tmp/dnsmasq.d/adb_list.winhelp:address=/ssl.google-analytics.com/192.0.2.1
-  /tmp/dnsmasq.d/adb_list.winhelp:address=/www.google-analytics.com/192.0.2.1
-  /tmp/dnsmasq.d/adb_list.yoyo:address=/google-analytics.com/192.0.2.1
+  /tmp/dnsmasq.d/adb_list.winhelp:address=/ssl.google-analytics.com/198.18.0.1
+  /tmp/dnsmasq.d/adb_list.winhelp:address=/www.google-analytics.com/198.18.0.1
+  /tmp/dnsmasq.d/adb_list.yoyo:address=/google-analytics.com/198.18.0.1
 </code></pre>
   
 **example to find blocked domains on certain sites for whitelisting:**
 <pre><code>
 1. the easy way ...
 enable the network analysis builtins in chrome or firefox to identify domains
-which are redirected to the adblock null-ip (default 192.0.2.1), add these domains to your whitelist
+which are redirected to the adblock null-ip (default 198.18.0.1), add these domains to your whitelist
 
 2. a bit harder ...
 enable 'Log queries' in the dnsmasq configuration (via LuCI Network => DHCP/DNS),
-ssh to your router and start tracing with 'logread -f -e "dnsmasq" -e "192.0.2.1"'
+ssh to your router and start tracing with 'logread -f -e "dnsmasq" -e "198.18.0.1"'
 switch to your client, access the relevant site and check all domains
 that are blocked/listed in logread, add these domains to your whitelist
 
@@ -222,10 +223,9 @@ If your awk one-liner works quite well, add a new source section in adblock conf
   
 ## Background
 This adblock package is a dns/dnsmasq based adblock solution.  
-Queries to ad/abuse domains are never forwarded and always replied with a local IP address which may be IPv4 or IPv6.  
-For that purpose adblock uses an ip address from the private 'TEST-NET-1' subnet (192.0.2.1 / ::ffff:c000:0201) by default (in AP mode the local router ip address will be used).  
-Furthermore all ad/abuse queries will be filtered by ip(6)tables and redirected to internal adblock pixel server (in PREROUTING chain) or rejected (in FORWARD or OUTPUT chain).  
-All iptables and uhttpd related adblock additions are non-destructive, no hard-coded changes in 'firewall.user', 'uhttpd' config or any other openwrt related config files. There is _no_ adblock background daemon running, the (scheduled) start of the adblock service keeps only the adblock lists up-to-date.  
+Queries to ad/abuse domains are never forwarded and always replied with a local IP address which may be IPv4 or IPv6. For that purpose adblock uses an ip address from the private 'Benchmark Test' subnet (198.18.0.1 / ::ffff:c612:0001) by default (in AP mode the local router ip address will be used). Furthermore all ad/abuse queries will be filtered by ip(6)tables and redirected to two uhttpd instances, separated for ads delivered on port 80 and on port 443 (in PREROUTING chain) or rejected (in FORWARD or OUTPUT chain).  
+  
+All iptables and uhttpd related adblock additions are non-destructive, no hard-coded changes in 'firewall.user', 'uhttpd' config or any other system related config files. There is _no_ adblock background daemon running, the (scheduled) start of the adblock service keeps only the adblock lists up-to-date.  
 
 ## Support
 Please join the adblock discussion in this [openwrt forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>  

net/adblock/files/adblock-helper.sh

@@ -32,11 +32,12 @@ f_envload()
     # may be overwritten by setting appropriate adblock config options in global section of /etc/config/adblock
     #
     adb_lanif="lan"
-    adb_nullport="65535"
-    adb_nullipv4="192.0.2.1"
-    adb_nullipv6="::ffff:c000:0201"
+    adb_nullport="65534"
+    adb_nullportssl="65535"
+    adb_nullipv4="198.18.0.1"
+    adb_nullipv6="::ffff:c612:0001"
     adb_whitelist="/etc/adblock/adblock.whitelist"
-    adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\$1)}"
+    adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}"
     adb_forcedns=1
     adb_fetchttl=5
     adb_restricted=0
@@ -120,7 +121,7 @@ f_envload()
     adb_fetch="$(which wget)"
     adb_uci="$(which uci)"
     adb_date="$(which date)"
-    unset adb_srclist adb_revsrclist adb_errsrclist
+    unset adb_srclist adb_revsrclist
 
     # check 'enabled' & 'version' config options
     #
@@ -140,26 +141,6 @@ f_envload()
         f_exit
     fi
 
-    # check running dnsmasq instance
-    #
-    check="$(pgrep -f "dnsmasq")"
-    if [ -z "${check}" ]
-    then
-        rc=-1
-        f_log "please enable the local dnsmasq instance to use adblock"
-        f_exit
-    fi
-
-    # check running firewall
-    #
-    check="$(${adb_iptv4} -vnL | grep -F "DROP")"
-    if [ -z "${check}" ]
-    then
-        rc=-1
-        f_log "please enable the local firewall to use adblock"
-        f_exit
-    fi
-
     # get lan ip addresses
     #
     network_get_ipaddr adb_ipv4 "${adb_lanif}"
@@ -189,15 +170,37 @@ f_envload()
     then
         adb_nullipv4="${adb_ipv4}"
         adb_nullipv6="${adb_ipv6}"
-        if [ "$(${adb_uci} get uhttpd.main.listen_http | grep -Fo "80")" = "80" ] ||
-           [ "$(${adb_uci} get uhttpd.main.listen_https | grep -Fo "443")" = "443" ]
+        if [ "$(${adb_uci} -q get uhttpd.main.listen_http | grep -Fo "80")" = "80" ] ||
+           [ "$(${adb_uci} -q get uhttpd.main.listen_https | grep -Fo "443")" = "443" ]
         then
             rc=-1
             f_log "AP mode detected, set local LuCI instance to ports <> 80/443"
             f_exit
+        elif [ -z "$(pgrep -f "dnsmasq")" ]
+        then
+            rc=-1
+            f_log "please enable the local dnsmasq instance to use adblock"
+            f_exit
+        elif [ -z "$(${adb_iptv4} -vnL | grep -Fo "DROP")" ]
+        then
+            rc=-1
+            f_log "please enable the local firewall to use adblock"
+            f_exit
         else
             apmode_ok="true"
         fi
+    else
+        check="$(${adb_uci} -q get bcp38.@bcp38[0].enabled)"
+        if [ $((check)) -eq 1 ]
+        then
+            check="$(${adb_uci} -q get bcp38.@bcp38[0].match | grep -Fo "${adb_nullipv4%.*}")"
+            if [ -n "${check}" ]
+            then
+                rc=-1
+                f_log "please whitelist '${adb_nullipv4}' in your bcp38 configuration to use default adblock null-ip"
+                f_exit
+            fi
+        fi
     fi
 
     # get system release level
@@ -349,16 +352,17 @@ f_envcheck()
     #
     if [ -n "${adb_wanif4}" ]
     then
-        f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-nat" "-p tcp -d ${adb_nullipv4} -m multiport --dports 80,443 -j DNAT --to-destination ${adb_ipv4}:${adb_nullport}"
-        f_firewall "IPv4" "filter" "A" "forwarding_rule" "adb-fwd" "-p tcp -d ${adb_nullipv4} -j REJECT --reject-with tcp-reset"
-        f_firewall "IPv4" "filter" "A" "forwarding_rule" "adb-fwd" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
-        f_firewall "IPv4" "filter" "A" "output_rule" "adb-out" "-p tcp -d ${adb_nullipv4} -j REJECT --reject-with tcp-reset"
-        f_firewall "IPv4" "filter" "A" "output_rule" "adb-out" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
         if [ $((adb_forcedns)) -eq 1 ] && [ -n "${adb_landev4}" ]
         then
-            f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-dns" "-i ${adb_landev4} -p udp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
-            f_firewall "IPv4" "nat" "A" "prerouting_rule" "adb-dns" "-i ${adb_landev4} -p tcp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
+            f_firewall "IPv4" "nat" "prerouting_rule" "prerouting_rule" "0" "dns" "-i ${adb_landev4} -p udp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
+            f_firewall "IPv4" "nat" "prerouting_rule" "prerouting_rule" "0" "dns" "-i ${adb_landev4} -p tcp --dport 53 -j DNAT --to-destination ${adb_ipv4}:53"
         fi
+        f_firewall "IPv4" "nat" "prerouting_rule" "adb-nat" "1" "nat" "-d ${adb_nullipv4} -p tcp --dport 80 -j DNAT --to-destination ${adb_ipv4}:${adb_nullport}"
+        f_firewall "IPv4" "nat" "prerouting_rule" "adb-nat" "2" "nat" "-d ${adb_nullipv4} -p tcp --dport 443 -j DNAT --to-destination ${adb_ipv4}:${adb_nullportssl}"
+        f_firewall "IPv4" "filter" "forwarding_rule" "adb-fwd" "1" "fwd" "-d ${adb_nullipv4} -p tcp -j REJECT --reject-with tcp-reset"
+        f_firewall "IPv4" "filter" "forwarding_rule" "adb-fwd" "2" "fwd" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
+        f_firewall "IPv4" "filter" "output_rule" "adb-out" "1" "out" "-d ${adb_nullipv4} -p tcp -j REJECT --reject-with tcp-reset"
+        f_firewall "IPv4" "filter" "output_rule" "adb-out" "2" "out" "-d ${adb_nullipv4} -j REJECT --reject-with icmp-host-unreachable"
         if [ "${fw_done}" = "true" ]
         then
             f_log "created volatile IPv4 firewall ruleset"
@@ -370,16 +374,17 @@ f_envcheck()
     #
     if [ -n "${adb_wanif6}" ]
     then
-        f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-nat" "-p tcp -d ${adb_nullipv6} -m multiport --dports 80,443 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullport}"
-        f_firewall "IPv6" "filter" "A" "forwarding_rule" "adb-fwd" "-p tcp -d ${adb_nullipv6} -j REJECT --reject-with tcp-reset"
-        f_firewall "IPv6" "filter" "A" "forwarding_rule" "adb-fwd" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
-        f_firewall "IPv6" "filter" "A" "output_rule" "adb-out" "-p tcp -d ${adb_nullipv6} -j REJECT --reject-with tcp-reset"
-        f_firewall "IPv6" "filter" "A" "output_rule" "adb-out" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
         if [ $((adb_forcedns)) -eq 1 ] && [ -n "${adb_landev6}" ]
         then
-            f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-dns" "-i ${adb_landev6} -p udp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
-            f_firewall "IPv6" "nat" "A" "PREROUTING" "adb-dns" "-i ${adb_landev6} -p tcp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
+            f_firewall "IPv6" "nat" "PREROUTING" "PREROUTING" "0" "dns" "-i ${adb_landev6} -p udp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
+            f_firewall "IPv6" "nat" "PREROUTING" "PREROUTING" "0" "dns" "-i ${adb_landev6} -p tcp --dport 53 -j DNAT --to-destination [${adb_ipv6}]:53"
         fi
+        f_firewall "IPv6" "nat" "PREROUTING" "adb-nat" "1" "nat" "-d ${adb_nullipv6} -p tcp --dport 80 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullport}"
+        f_firewall "IPv6" "nat" "PREROUTING" "adb-nat" "2" "nat" "-d ${adb_nullipv6} -p tcp --dport 443 -j DNAT --to-destination [${adb_ipv6}]:${adb_nullportssl}"
+        f_firewall "IPv6" "filter" "forwarding_rule" "adb-fwd" "1" "fwd" "-d ${adb_nullipv6} -p tcp -j REJECT --reject-with tcp-reset"
+        f_firewall "IPv6" "filter" "forwarding_rule" "adb-fwd" "2" "fwd" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
+        f_firewall "IPv6" "filter" "output_rule" "adb-out" "1" "out" "-d ${adb_nullipv6} -p tcp -j REJECT --reject-with tcp-reset"
+        f_firewall "IPv6" "filter" "output_rule" "adb-out" "2" "out" "-d ${adb_nullipv6} -j REJECT --reject-with icmp6-addr-unreachable"
         if [ "${fw_done}" = "true" ]
         then
             f_log "created volatile IPv6 firewall ruleset"
@@ -394,22 +399,23 @@ f_envcheck()
     then
         if [ -n "${adb_wanif4}" ] && [ -n "${adb_wanif6}" ]
         then
-            uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}" -p "[${adb_ipv6}]:${adb_nullport}"
-            rc=${?}
+            uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}" -p "[${adb_ipv6}]:${adb_nullport}"
+            uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullportssl}" -p "[${adb_ipv6}]:${adb_nullportssl}"
         elif [ -n "${adb_wanif4}" ]
         then
-            uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}"
-            rc=${?}
+            uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullport}"
+            uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "${adb_ipv4}:${adb_nullportssl}"
         else
-            uhttpd -h "/www/adblock" -k 0 -N 100 -t 0 -T 1 -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullport}"
-            rc=${?}
+            uhttpd -h "/www/adblock" -N 25 -T 1 -k 0 -t 0 -R -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullport}"
+            uhttpd -h "/www/adblock" -N 25 -T 0 -k 0 -t 0 -R -D -S -E "/index.html" -p "[${adb_ipv6}]:${adb_nullportssl}"
         fi
+        rc=${?}
         if [ $((rc)) -eq 0 ]
         then
-            f_log "created volatile uhttpd instance"
+            f_log "created volatile uhttpd instances"
         else
-            f_log "failed to initialize volatile uhttpd instance" "${rc}"
-            f_restore
+            f_log "failed to initialize volatile uhttpd instances" "${rc}"
+            f_exit
         fi
     fi
 
@@ -446,26 +452,44 @@ f_depend()
 f_firewall()
 {
     local ipt="${adb_iptv4}"
+    local nullip="${adb_nullipv4}"
     local proto="${1}"
     local table="${2}"
-    local ctype="${3}"
+    local chsrc="${3}"
     local chain="${4}"
-    local notes="${5}"
-    local rules="${6}"
+    local chpos="${5}"
+    local notes="adb-${6}"
+    local rules="${7}"
 
     # select appropriate iptables executable for IPv6
     #
     if [ "${proto}" = "IPv6" ]
     then
         ipt="${adb_iptv6}"
+        nullip="${adb_nullipv6}"
     fi
 
-    # check whether iptables rule already applied and proceed accordingly
+    # check whether iptables chain already exist
     #
-    rc="$("${ipt}" -w -t "${table}" -C "${chain}" -m comment --comment "${notes}" ${rules}; printf ${?})"
+    rc="$("${ipt}" -w -t "${table}" -nL "${chain}" >/dev/null 2>&1; printf ${?})"
     if [ $((rc)) -ne 0 ]
     then
-        "${ipt}" -w -t "${table}" -"${ctype}" "${chain}" -m comment --comment "${notes}" ${rules}
+        "${ipt}" -w -t "${table}" -N "${chain}"
+        "${ipt}" -w -t "${table}" -A "${chain}" -m comment --comment "${notes}" -j RETURN
+        "${ipt}" -w -t "${table}" -A "${chsrc}" -d "${nullip}" -m comment --comment "${notes}" -j "${chain}"
+    fi
+
+    # check whether iptables rule already exist
+    #
+    rc="$("${ipt}" -w -t "${table}" -C "${chain}" -m comment --comment "${notes}" ${rules} >/dev/null 2>&1; printf ${?})"
+    if [ $((rc)) -ne 0 ]
+    then
+        if [ $((chpos)) -eq 0 ]
+        then
+            "${ipt}" -w -t "${table}" -A "${chain}" -m comment --comment "${notes}" ${rules}
+        else
+            "${ipt}" -w -t "${table}" -I "${chain}" "${chpos}" -m comment --comment "${notes}" ${rules}
+        fi
         rc=${?}
         if [ $((rc)) -eq 0 ]
         then
@@ -552,91 +576,21 @@ f_rmconfig()
 {
     local src_name
     local rm_done="${1}"
-    local restore_done="${2:-false}"
-
+ 
     for src_name in ${rm_done}
     do
         src_name="${src_name#*.}"
-        if [ "${restore_done}" = "true" ]
-        then
-            src_name="${src_name%.*}"
-            "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=list restored"
-        else
-            "${adb_uci}" -q delete "adblock.${src_name}.adb_src_count"
-            "${adb_uci}" -q delete "adblock.${src_name}.adb_src_timestamp"
-        fi
+        "${adb_uci}" -q delete "adblock.${src_name}.adb_src_count"
+        "${adb_uci}" -q delete "adblock.${src_name}.adb_src_timestamp"
     done
 }
 
-# f_restore: restore last adblock list backups and restart dnsmasq
-#
-f_restore()
-{
-    local rm_done
-    local restore_done
-
-    # remove bogus adblock lists
-    #
-    if [ -n "${adb_revsrclist}" ]
-    then
-        rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec rm -f "{}" \;)"
-        rc=${?}
-        if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
-        then
-            f_rmconfig "${rm_done}"
-            f_log "all bogus adblock lists removed"
-        elif [ $((rc)) -ne 0 ]
-        then
-            f_log "error during removal of bogus adblock lists" "${rc}"
-            f_exit
-        fi
-    fi
-
-    # restore backups
-    #
-    if [ "${backup_ok}" = "true" ]
-    then
-        restore_done="$(find "${adb_dir_backup}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -print -exec cp -pf "{}" "${adb_dnsdir}" \;)"
-        rc=${?}
-        if [ $((rc)) -eq 0 ] && [ -n "${restore_done}" ]
-        then
-            find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -exec gunzip -f "{}" \;
-            f_log "all available backups restored"
-        elif [ $((rc)) -ne 0 ] && [ -n "${restore_done}" ]
-        then
-            f_log "error during restore of adblock lists" "${rc}"
-            f_exit
-        fi
-    else
-        f_log "backup service disabled, nothing to restore"
-    fi
-
-    # (re-)try dnsmasq restart without bogus adblock lists / with backups 
-    #
-    if [ -n "${restore_done}" ] || [ -n "${rm_done}" ]
-    then
-        /etc/init.d/dnsmasq restart
-        sleep 1
-        check="$(pgrep -f "dnsmasq")"
-        if [ -n "${check}" ]
-        then
-            rc=0
-            f_cntconfig
-            f_log "adblock lists with overall ${adb_count} domains loaded"
-        else
-            rc=120
-            f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
-        fi
-    fi
-    f_exit
-}
-
 # f_exit: delete (temporary) files, generate statistics and exit
 #
 f_exit()
 {
-    local ipv4_adblock=0
-    local ipv6_adblock=0
+    local ipv4_blk=0 ipv4_all=0 ipv4_pct=0
+    local ipv6_blk=0 ipv6_all=0 ipv6_pct=0
     local lastrun="$(${adb_date} "+%d.%m.%Y %H:%M:%S")"
 
     # delete temporary files & directories
@@ -656,19 +610,26 @@ f_exit()
     then
         if [ -n "${adb_wanif4}" ]
         then
-            ipv4_adblock="$(${adb_iptv4} -t nat -vnL | awk '$11 ~ /^adb-nat$/ {sum += $1} END {printf sum}')"
-            ipv4_adblock="$((${ipv4_adblock} + $(${adb_iptv4} -vnL | awk '$11 ~ /^adb-(fwd|out)$/ {sum += $1} END {printf sum}')))"
+            ipv4_blk="$(${adb_iptv4} -t nat -vnL adb-nat | awk '$3 ~ /^DNAT$/ {sum += $1} END {printf sum}')"
+            ipv4_all="$(${adb_iptv4} -t nat -vnL PREROUTING | awk '$3 ~ /^prerouting_rule$/ {sum += $1} END {printf sum}')"
+            if [ $((ipv4_all)) -gt 0 ] && [ $((ipv4_blk)) -gt 0 ] && [ $((ipv4_all)) -gt $((ipv4_blk)) ]
+            then
+                ipv4_pct="$(printf "${ipv4_blk}" | awk -v all="${ipv4_all}" '{printf( "%5.2f\n",$1/all*100)}')"
+            fi
         fi
         if [ -n "${adb_wanif6}" ]
         then
-            ipv6_adblock="$(${adb_iptv6} -t nat -vnL | awk '$10 ~ /^adb-nat$/ {sum += $1} END {printf sum}')"
-            ipv6_adblock="$((${ipv6_adblock} + $(${adb_iptv6} -vnL | awk '$10 ~ /^adb-(fwd|out)$/ {sum += $1} END {printf sum}')))"
-        fi
-        if [ -n "$(${adb_uci} -q changes adblock)" ]
-        then
-            "${adb_uci}" -q commit "adblock"
+            ipv6_blk="$(${adb_iptv6} -t nat -vnL adb-nat | awk '$3 ~ /^DNAT$/ {sum += $1} END {printf sum}')"
+            ipv6_all="$(${adb_iptv6} -t nat -vnL PREROUTING | awk '$3 ~ /^(adb-nat|DNAT)$/ {sum += $1} END {printf sum}')"
+            if [ $((ipv6_all)) -gt 0 ] && [ $((ipv6_blk)) -gt 0 ] && [ $((ipv6_all)) -gt $((ipv6_blk)) ]
+            then
+                ipv6_pct="$(printf "${ipv6_blk}" | awk -v all="${ipv6_all}" '{printf( "%5.2f\n",$1/all*100)}')"
+            fi
         fi
-        f_log "firewall statistics (IPv4/IPv6): ${ipv4_adblock}/${ipv6_adblock} ad related packets blocked"
+        "${adb_uci}" -q set "adblock.global.adb_percentage=${ipv4_pct}%/${ipv6_pct}%"
+        "${adb_uci}" -q set "adblock.global.adb_lastrun=${lastrun}"
+        "${adb_uci}" -q commit "adblock"
+        f_log "firewall statistics (IPv4/IPv6): ${ipv4_pct}%/${ipv6_pct}% of all packets in prerouting chain are ad related & blocked"
         f_log "domain adblock processing finished successfully (${adb_scriptver}, ${adb_sysver}, ${lastrun})"
     elif [ $((rc)) -gt 0 ]
     then

net/adblock/files/adblock-update.sh

@@ -28,8 +28,8 @@ fi
 # get current directory and set script/config version
 #
 adb_scriptdir="${0%/*}"
-adb_scriptver="1.1.17"
-adb_mincfgver="2.1"
+adb_scriptver="1.2.0"
+adb_mincfgver="2.2"
 
 # source in adblock function library
 #
@@ -88,7 +88,7 @@ do
         adb_srclist="${adb_srclist} -a ! -name ${adb_dnsprefix}.${src_name}*"
     fi
 
-    # only download adblock list with newer/updated timestamp
+    # download only block list with newer/updated timestamp
     #
     if [ "${src_name}" = "blacklist" ]
     then
@@ -99,7 +99,7 @@ do
     if [ -z "${url_time}" ]
     then
         url_time="$(date)"
-        f_log "   no online timestamp received, current date will be used"
+        f_log "   online timestamp not received, current date will be used"
     fi
     if [ -z "${list_time}" ] || [ "${list_time}" != "${url_time}" ] || [ ! -r "${adb_dnsfile}" ] ||\
       ([ "${backup_ok}" = "true" ] && [ ! -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ])
@@ -107,7 +107,6 @@ do
         if [ "${src_name}" = "blacklist" ]
         then
             tmp_domains="$(cat "${url}")"
-            rc=${?}
         elif [ "${src_name}" = "shalla" ]
         then
             shalla_archive="${adb_tmpdir}/shallalist.tar.gz"
@@ -127,30 +126,31 @@ do
                         break
                     fi
                 done
-                rm -f "${shalla_archive}"
-                rm -rf "${adb_tmpdir}/BL"
                 tmp_domains="$(cat "${shalla_file}")"
-                rc=${?}
+                rm -rf "${adb_tmpdir}/BL"
+                rm -f "${shalla_archive}"
+                rm -f "${shalla_file}"
             fi
         else
             tmp_domains="$(${adb_fetch} ${fetch_parm} --output-document=- "${url}")"
-            rc=${?}
         fi
+        rc=${?}
     else
         f_log "   source doesn't change, no update required"
         continue
     fi
 
-    # check download result and prepare domain output by regex patterns
+    # check download result and prepare domain output, backup/restore if needed
     #
     if [ $((rc)) -eq 0 ] && [ -n "${tmp_domains}" ]
     then
         count="$(printf "%s\n" "${tmp_domains}" | awk "${src_rset}" | tee "${adb_tmpfile}" | wc -l)"
-        f_log "   source download finished (${count} entries)"
-        if [ "${src_name}" = "shalla" ]
+        "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=${url_time}"
+        if [ "${backup_ok}" = "true" ]
         then
-            rm -f "${shalla_file}"
+            gzip -cf "${adb_tmpfile}" > "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
         fi
+        f_log "   source download finished (${count} entries)"
         unset tmp_domains
     elif [ $((rc)) -eq 0 ] && [ -z "${tmp_domains}" ]
     then
@@ -159,53 +159,61 @@ do
         continue
     else
         rc=0
-        if [ -z "${adb_errsrclist}" ]
+        if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
         then
-            adb_errsrclist="-name ${adb_dnsprefix}.${src_name}.gz"
+            gunzip -cf "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" > "${adb_tmpfile}"
+            count="$(wc -l < "${adb_tmpfile}")"
+            "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=list restored"
+            f_log "   source download failed, list restored (${count} entries)"
         else
-            adb_errsrclist="${adb_errsrclist} -o -name ${adb_dnsprefix}.${src_name}.gz"
+            "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=download failed"
+            f_log "   source download failed"
+            continue
         fi
-        "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=download failed"
-        f_log "   source download failed"
-        continue
     fi
 
     # remove whitelist domains, sort domains and make them unique,
-    # finally rewrite ad/abuse domain information to separate dnsmasq files
+    # rewrite ad/abuse domain information to separate dnsmasq files
     #
     if [ $((count)) -gt 0 ] && [ -n "${adb_tmpfile}" ]
     then
         if [ -s "${adb_tmpdir}/tmp.whitelist" ]
         then
             grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" | sort -u | eval "${adb_dnsformat}" > "${adb_dnsfile}"
-            rc=${?}
         else
             sort -u "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}"
-            rc=${?}
         fi
+        rc=${?}
 
-        # prepare find statement with revised adblock list sources
-        #
-        if [ -z "${adb_revsrclist}" ]
-        then
-            adb_revsrclist="-name ${adb_dnsprefix}.${src_name}"
-        else
-            adb_revsrclist="${adb_revsrclist} -o -name ${adb_dnsprefix}.${src_name}"
-        fi
-
-        # store source timestamp in config
+        # finish domain processing, prepare find statement with revised adblock list source
         #
         if [ $((rc)) -eq 0 ]
         then
-            "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=${url_time}"
+            if [ -z "${adb_revsrclist}" ]
+            then
+                adb_revsrclist="-name ${adb_dnsprefix}.${src_name}"
+            else
+                adb_revsrclist="${adb_revsrclist} -o -name ${adb_dnsprefix}.${src_name}"
+            fi
             f_log "   domain merging finished"
         else
-            f_log "   domain merging failed" "${rc}"
-            f_restore
+            rc=0
+            rm -f "${adb_dnsfile}"
+            if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
+            then
+                rm -f "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
+            fi
+            f_log "   domain merging failed, list removed"
+            continue
         fi
     else
+        rm -f "${adb_dnsfile}"
+        if [ "${backup_ok}" = "true" ] && [ -r "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz" ]
+        then
+            rm -f "${adb_dir_backup}/${adb_dnsprefix}.${src_name}.gz"
+        fi
         "${adb_uci}" -q set "adblock.${src_name}.adb_src_timestamp=empty domain input"
-        f_log "   empty domain input received"
+        f_log "   empty domain input received, list removed"
         continue
     fi
 done
@@ -215,45 +223,21 @@ done
 if [ -n "${adb_srclist}" ]
 then
     rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_srclist} \) -print -exec rm -f "{}" \;)"
-    rc=${?}
     if [ "${backup_ok}" = "true" ] && [ -n "${rm_done}" ]
     then
         find "${adb_dir_backup}" -maxdepth 1 -type f \( ${adb_srclist} \) -exec rm -f "{}" \;
     fi
 else
     rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
-    rc=${?}
     if [ "${backup_ok}" = "true" ]
     then
         find "${adb_dir_backup}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -exec rm -f "{}" \;
     fi
 fi
-if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
+if [ -n "${rm_done}" ]
 then
     f_rmconfig "${rm_done}"
-    f_log "remove disabled adblock lists"
-elif [ $((rc)) -ne 0 ] && [ -n "${rm_done}" ]
-then
-    f_log "error during removal of disabled adblock lists" "${rc}"
-    f_exit
-fi
-
-# partial restore of adblock lists in case of download errors
-#
-if [ "${backup_ok}" = "true" ] && [ -n "${adb_errsrclist}" ]
-then
-    restore_done="$(find "${adb_dir_backup}" -maxdepth 1 -type f \( ${adb_errsrclist} \) -print -exec cp -pf "{}" "${adb_dnsdir}" \;)"
-    rc=${?}
-    if [ $((rc)) -eq 0 ] && [ -n "${restore_done}" ]
-    then
-        find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*.gz" -exec gunzip -f "{}" \;
-        f_rmconfig "${restore_done}" "true"
-        f_log "partial restore done"
-    elif [ $((rc)) -ne 0 ]
-    then
-        f_log "error during partial restore" "${rc}"
-        f_exit
-    fi
+    f_log "disabled adblock lists removed"
 fi
 
 # make separate adblock lists entries unique
@@ -289,7 +273,7 @@ fi
 #
 f_cntconfig
 adb_count="$(${adb_uci} -q get "adblock.global.adb_overall_count")"
-if [ -n "${adb_revsrclist}" ] || [ -n "${rm_done}" ] || [ -n "${restore_done}" ] || [ -n "${mv_done}" ]
+if [ -n "${adb_revsrclist}" ] || [ -n "${rm_done}" ]
 then
     "${adb_uci}" -q set "adblock.global.adb_dnstoggle=on"
     /etc/init.d/dnsmasq restart
@@ -299,31 +283,30 @@ then
     then
         f_log "adblock lists with overall ${adb_count} domains loaded"
     else
-        rc=100
-        f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
-        f_restore
+        f_log "dnsmasq restart failed, retry without newly generated block lists"
+        rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec rm -f "{}" \;)"
+        if [ -n "${rm_done}" ]
+        then
+            f_log "bogus adblock lists removed"
+            f_rmconfig "${rm_done}"
+            /etc/init.d/dnsmasq restart
+            sleep 1
+            check="$(pgrep -f "dnsmasq")"
+            if [ -n "${check}" ]
+            then
+                f_cntconfig
+                f_log "adblock lists with overall ${adb_count} domains loaded"
+            else
+                rc=100
+                f_log "dnsmasq restart failed, please check 'logread' output" "${rc}"
+                f_exit
+            fi
+        fi
     fi
 else
     f_log "adblock lists with overall ${adb_count} domains are still valid, no update required"
 fi
 
-# create adblock list backups
-#
-if [ "${backup_ok}" = "true" ] && [ -n "${adb_revsrclist}" ]
-then
-    backup_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f \( ${adb_revsrclist} \) -print -exec cp -pf "{}" "${adb_dir_backup}" \;)"
-    rc=${?}
-    if [ $((rc)) -eq 0 ] && [ -n "${backup_done}" ]
-    then
-        find "${adb_dir_backup}" -maxdepth 1 -type f \( -name "${adb_dnsprefix}*" -a ! -name "${adb_dnsprefix}*.gz" \) -exec gzip -f "{}" \;
-        f_log "new adblock list backups generated"
-    elif [ $((rc)) -ne 0 ] && [ -n "${backup_done}" ]
-    then
-        f_log "error during backup of adblock lists" "${rc}"
-        f_exit
-    fi
-fi
-
 # remove temporary files and exit
 #
 f_exit

net/adblock/files/adblock.conf

@@ -3,9 +3,9 @@
 
 config adblock 'global'
 	option adb_enabled '1'
-	option adb_cfgver '2.1'
+	option adb_cfgver '2.2'
 	option adb_whitelist '/etc/adblock/adblock.whitelist'
-	option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\$1)}'
+	option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}'
 	option adb_forcedns '1'
 
 config service 'backup'

net/adblock/files/adblock.init

@@ -59,45 +59,65 @@ start()
 
 restart()
 {
+    restart="true"
     stop
     start
 }
 
 reload()
 {
+    restart="true"
     stop
     start
 }
 
 stop()
 {
-    if [ -n "$(${adb_uci} -q get "adblock.global.adb_overall_count")" ]
+    cfg_check="$(${adb_uci} -q get "adblock.global.adb_overall_count")"
+    if [ -n "${cfg_check}" ]
     then
         . "/lib/functions.sh"
         config_load adblock
         config_foreach rm_config source
         "${adb_uci}" -q delete "adblock.global.adb_dnstoggle"
         "${adb_uci}" -q delete "adblock.global.adb_overall_count"
-        "${adb_uci}" -q commit "adblock"
+        "${adb_uci}" -q delete "adblock.global.adb_percentage"
+        "${adb_uci}" -q delete "adblock.global.adb_lastrun"
+        if [ -z "${restart}" ]
+        then
+            "${adb_uci}" -q commit "adblock"
+        fi
+    fi
+    fw_check="$(iptables -w -t nat -nL | grep -F "adb-")"
+    if [ -n "${fw_check}" ] && [ -z "${restart}" ]
+    then
+        iptables-save -t nat | grep -Fv -- "adb-" | iptables-restore
+        iptables-save -t filter | grep -Fv -- "adb-" | iptables-restore
+        if [ -n "$(lsmod | grep -F "ip6table_nat")" ]
+        then
+            ip6tables-save -t nat | grep -Fv -- "adb-" | ip6tables-restore
+            ip6tables-save -t filter | grep -Fv -- "adb-" | ip6tables-restore
+        fi
     fi
     if [ -d "${adb_dnshidedir}" ]
     then
         find "${adb_dnshidedir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -exec mv -f "{}" "${adb_dnsdir}" \;
     fi
-    rm_done="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
-    rc=${?}
-    if [ $((rc)) -eq 0 ] && [ -n "${rm_done}" ]
+    dns_check="$(find "${adb_dnsdir}" -maxdepth 1 -type f -name "${adb_dnsprefix}*" -print -exec rm -f "{}" \;)"
+    if [ -n "${dns_check}" ]
     then
         rm -rf "${adb_dnshidedir}"
         /etc/init.d/dnsmasq restart
-        /etc/init.d/firewall restart
     fi
-    uhttpd_pid="$(pgrep -f "uhttpd -h /www/adblock")"
-    if [ -n "${uhttpd_pid}" ]
+    www_check="$(pgrep -f "uhttpd -h /www/adblock")"
+    if [ -n "${www_check}" ]
     then
-        kill -9 "${uhttpd_pid}"
+        for pid in ${www_check}
+        do
+            kill -9 "${pid}"
+        done
     fi
-    if [ -n "${rm_done}" ] || [ -n "${uhttpd_pid}" ]
+    if [ -n "${cfg_check}" ] || [ -n "${fw_check}" ] || [ -n "${dns_check}" ] || [ -n "${www_check}" ]
     then
         "${adb_log}" ${log_parm} -t "adblock[${adb_pid}] info " "all adblock related services stopped" 2>&1
     fi