adblock: update 2.5.0

* revamp LuCI GUI (see luci repo for details)
* integrated runtime statistics in normal adblock ubus service instance
(see readme)
* simplified boot/init
    * no longer use raw interface trigger
    * use only network interface trigger, pre-configured for
'wan/wwan/lan' interfaces (see readme)
    * fixed a reload issue
* removed switch to disable tld compression (seems to be pretty stable)
* muted awk parser if source list contains incorrect records
* raise dns backend timeout to 20 seconds
* documentation update
* cosmetics

Signed-off-by: Dirk Brenken <dev@brenken.org>

net/adblock/Makefile

@@ -6,7 +6,7 @@
 include $(TOPDIR)/rules.mk
 
 PKG_NAME:=adblock
-PKG_VERSION:=2.4.0
+PKG_VERSION:=2.5.0
 PKG_RELEASE:=2
 PKG_LICENSE:=GPL-3.0+
 PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org>

net/adblock/files/README.md

@@ -64,8 +64,8 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
 * quality checks during block list update to ensure a reliable dns backend service
 * minimal status & error logging to syslog, enable debug logging to receive more output
-* procd based init system support (start/stop/restart/reload/suspend/resume)
-* procd based hotplug support, the adblock start will be triggered by interface triggers
+* procd based init system support (start/stop/restart/reload/suspend/resume/query)
+* procd based hotplug support, the adblock start will be solely triggered by network interface triggers
 * suspend & resume adblock actions temporarily without block list reloading
 * runtime statistics via ubus service call
 * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
@@ -73,8 +73,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * optional: add new adblock sources on your own via uci config
 
 ## Prerequisites
-* [openwrt](https://openwrt.org), tested with latest stable release (Chaos Calmer) and with current trunk (Designated Driver)
-* [LEDE project](https://www.lede-project.org), tested with trunk > r98
+* [LEDE project](https://www.lede-project.org), tested with latest stable release (LEDE 17.01) and with current LEDE snapshot
 * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
 * a download utility: full versions (with ssl support) of 'wget', 'uclient-fetch', 'aria2c' or 'curl' are supported - the Chaos Calmer built-in busybox wget is not
     * Chaos Calmer: download & install the external 'wget' package
@@ -87,19 +86,11 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
 
 ## LuCI adblock companion package
-* for easy management of the various block list sources and options you can also use a nice & efficient LuCI frontend
+* for easy management of the various block list sources and all other adblock options you can also use a nice & efficient LuCI frontend
 * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
 * the application is located in LuCI under 'Services' menu
 * _Thanks to Hannu Nyman for this great adblock LuCI frontend!_
 
-## Chaos Calmer installation notes
-* 'adblock' and 'luci-app-adblock' are _not_ available as .ipk packages in the Chaos Calmer download repository
-* download both packages from a development snapshot package directory:
-    * for 'adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/packages/)
-    * for 'luci-app-adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/luci/)
-* manually transfer the packages to your routers temp directory (with tools like _sshfs_ or _winscp_)
-* install the packages with _opkg install <...>_ as described above
-
 ## Tweaks
 * **status/runtime statistics:** the adblock status and runtime statistics are available via ubus service call (see example below)
 * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
@@ -118,10 +109,9 @@ A lot of people already use adblocker plugins within their desktop browsers, but
 * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
     * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
     * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
-    * adb\_iface => restrict the procd interface trigger to a (list of) certain wan interface(s) or disable it at all (default: not set, disabled)
+    * adb\_iface => set the procd interface trigger to a (list of) lan/wan interface(s) (default: 'wan wwan lan')
     * adb\_fetch => full path to a different download utility, see example below (default: not set, use wget)
     * adb\_fetchparm => options for the download utility, see example below (default: not set, use wget options)
-    * adb\_tldcomp => enable/disable tld compression (default: '1', enabled)
 
 ## Examples
 
@@ -155,50 +145,42 @@ curl:
   
 **receive adblock statistics via ubus:**
 <pre><code>
-ubus call service list '{"name":"adblock_stats"}'
+ubus call service get_data '{"name":"adblock"}
 This will output the active block lists and other runtime information as JSON, e.g.:
 {
-    "adblock_stats": {
-        "instances": {
-            "statistics": {
-                "running": false,
-                "command": [
-                    ""
-                ],
-                "data": {
-                    "active_lists": [
-                        {
-                            "palevo": "14",
-                            "blacklist": "144",
-                            "winspy": "168",
-                            "zeus": "422",
-                            "adaway": "408",
-                            "rolist": "649",
-                            "malwarelist": "1219",
-                            "ransomware": "1495",
-                            "ruadlist": "1791",
-                            "yoyo": "2304",
-                            "openphish": "2139",
-                            "dshield": "154",
-                            "disconnect": "3176",
-                            "spam404": "6251",
-                            "adguard": "11081",
-                            "whocares": "11575",
-                            "winhelp": "10574",
-                            "malware": "13854",
-                            "sysctl": "8539",
-                            "securemecca": "9262",
-                            "shalla": "25358",
-                            "hphosts": "36256"
-                        }
-                    ],
-                    "adblock_version": "2.3.0",
-                    "blocked_domains": "146833",
-                    "dns_backend": "dnsmasq",
-                    "last_rundate": "04.02.2017 21:10:31",
-                    "system": "LEDE Reboot SNAPSHOT r3286-c980147527"
+    "adblock": {
+        "adblock": {
+            "active_lists": [
+                {
+                    "palevo": "14",
+                    "blacklist": "147",
+                    "winspy": "138",
+                    "adaway": "378",
+                    "zeus": "397",
+                    "rolist": "652",
+                    "malwarelist": "1157",
+                    "yoyo": "2272",
+                    "ruadlist": "1793",
+                    "ransomware": "1670",
+                    "dshield": "190",
+                    "openphish": "2672",
+                    "disconnect": "2910",
+                    "spam404": "5715",
+                    "whocares": "8900",
+                    "winhelp": "8528",
+                    "adguard": "10790",
+                    "securemecca": "5080",
+                    "sysctl": "7906",
+                    "malware": "14617",
+                    "hphosts": "12450",
+                    "shalla": "23926"
                 }
-            }
+            ],
+            "adblock_version": "2.5.0",
+            "blocked_domains": "112302",
+            "dns_backend": "dnsmasq",
+            "last_rundate": "25.03.2017 21:09:12",
+            "system": "LEDE Reboot SNAPSHOT r3867-313197d707"
         }
     }
 }
@@ -295,7 +277,7 @@ If your awk one-liner works quite well, add a new source section in adblock conf
 </code></pre>
   
 ## Support
-Please join the adblock discussion in this [forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>  
+Please join the adblock discussion in this [forum thread](https://forum.lede-project.org/t/adblock-2-x-support-thread/507) or contact me by mail <dev@brenken.org>  
 
 ## Removal
 * stop all adblock related services with _/etc/init.d/adblock stop_

net/adblock/files/adblock.conf

@@ -4,6 +4,7 @@
 config adblock 'global'
 	option adb_enabled '1'
 	option adb_debug '0'
+	option adb_iface 'wan wwan lan'
 	option adb_whitelist '/etc/adblock/adblock.whitelist'
 	option adb_whitelist_rset '\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}'
 	option adb_backup '0'

net/adblock/files/adblock.init

@@ -9,32 +9,24 @@ EXTRA_HELP="	suspend	Suspend adblock processing
 	resume	Resume adblock processing
 	query	<DOMAIN> Query active blocklists for specific domain"
 
+adb_init="/etc/init.d/adblock"
 adb_script="/usr/bin/adblock.sh"
 
 boot()
 {
-    local wanif4 wanif6 cnt=0 max_cnt=30
+    local list iface="$(uci -q get adblock.global.adb_iface)"
 
-    . "/lib/functions/network.sh"
-    while [ ${cnt} -le ${max_cnt} ]
+    for name in ${iface}
     do
-        network_find_wan wanif4
-        network_find_wan6 wanif6
-        if [ -n "${wanif4}" ] || [ -n "${wanif6}" ] || [ ${cnt} -eq ${max_cnt} ]
-        then
-            rc_procd start_service
-            return 0
-        else
-            sleep 1
-            cnt=$((cnt+1))
-            network_flush_cache
-        fi
+        list="${list} network.interface.${name}"
     done
+    ubus -t 60 wait_for network.interface ${list}
+    rc_procd start_service
 }
 
 start_service()
 {
-    if [ $(/etc/init.d/adblock enabled; printf ${?}) -eq 0 ]
+    if [ $("${adb_init}" enabled; printf ${?}) -eq 0 ]
     then
         procd_open_instance "adblock"
         procd_set_param command "${adb_script}" "${@}"
@@ -44,11 +36,6 @@ start_service()
     fi
 }
 
-reload_service()
-{
-    rc_procd start_service reload
-}
-
 stop_service()
 {
     rc_procd "${adb_script}" stop
@@ -61,12 +48,12 @@ restart()
 
 suspend()
 {
-    rc_procd start_service suspend
+    rc_procd "${adb_script}" suspend
 }
 
 resume()
 {
-    rc_procd start_service resume
+    rc_procd "${adb_script}" resume
 }
 
 query()
@@ -78,14 +65,9 @@ service_triggers()
 {
     local iface="$(uci -q get adblock.global.adb_iface)"
 
-    if [ -z "${iface}" ]
-    then
-        procd_add_raw_trigger "interface.*.up" 1000 /etc/init.d/adblock start
-    else
-        for name in ${iface}
-        do
-            procd_add_interface_trigger "interface.*.up" "${name}" /etc/init.d/adblock start
-        done
-    fi
-    procd_add_config_trigger "config.change" "adblock" /etc/init.d/adblock start
+    for name in ${iface}
+    do
+        procd_add_interface_trigger "interface.*.up" "${name}" "${adb_init}" start
+    done
+    procd_add_config_trigger "config.change" "adblock" "${adb_init}" start
 }

net/adblock/files/adblock.sh

@@ -10,11 +10,11 @@
 #
 LC_ALL=C
 PATH="/usr/sbin:/usr/bin:/sbin:/bin"
-adb_ver="2.4.0-2"
+adb_ver="2.5.0"
+adb_sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')"
 adb_enabled=1
 adb_debug=0
 adb_backup=0
-adb_tldcomp=1
 adb_backupdir="/mnt"
 adb_whitelist="/etc/adblock/adblock.whitelist"
 adb_whitelist_rset="\$1 ~/^([A-Za-z0-9_-]+\.){1,}[A-Za-z]+/{print tolower(\"^\"\$1\"\\\|[.]\"\$1)}"
@@ -40,7 +40,7 @@ f_envload()
 
     # set dns backend environment
     #
-    while [ ${cnt} -le 10 ]
+    while [ ${cnt} -le 20 ]
     do
         for dns in ${adb_dnslist}
         do
@@ -188,7 +188,6 @@ f_rmdns()
         rm -f "${adb_backupdir}/${adb_dnsprefix}"*.gz
         rm -rf "${adb_dnshidedir}"
     fi
-    ubus call service delete "{\"name\":\"adblock_stats\",\"instances\":\"statistics\"}" 2>/dev/null
 }
 
 # f_dnsrestart: restart the dns backend
@@ -203,7 +202,7 @@ f_dnsrestart()
         adb_dnsup="$(ubus -S call service list "{\"name\":\"${adb_dns}\"}" | jsonfilter -l1 -e "@.${adb_dns}.instances.*.running")"
         if [ "${adb_dnsup}" = "true" ]
         then
-            return 0
+            break
         fi
         cnt=$((cnt+1))
         sleep 1
@@ -323,7 +322,7 @@ f_log()
         logger -t "adblock-[${adb_ver}] ${class}" "${log_msg}"
         if [ "${class}" = "error" ]
         then
-            logger -t "adblock-[${adb_ver}] ${class}" "Please check the online documentation 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md'"
+            logger -t "adblock-[${adb_ver}] ${class}" "Please check 'https://github.com/openwrt/packages/blob/master/net/adblock/files/README.md' (${adb_sysver})"
             f_rmtemp
             if [ "$(ls -dA "${adb_dnsdir}/${adb_dnsprefix}"* >/dev/null 2>&1)" ]
             then
@@ -340,8 +339,7 @@ f_log()
 f_main()
 {
     local enabled url cnt sum_cnt=0 mem_total=0
-    local src_name src_rset shalla_archive list active_lists
-    local sysver="$(ubus -S call system board | jsonfilter -e '@.release.description')"
+    local src_name src_rset shalla_archive list active_lists active_triggers
     mem_total="$(awk '$1 ~ /^MemTotal/ {printf $2}' "/proc/meminfo" 2>/dev/null)"
 
     f_log "info " "start adblock processing ..."
@@ -398,17 +396,12 @@ f_main()
         #
         if [ ${adb_rc} -eq 0 ] && [ -s "${adb_tmpload}" ]
         then
-            awk "${src_rset}" "${adb_tmpload}" > "${adb_tmpfile}"
+            awk "${src_rset}" "${adb_tmpload}" 2>/dev/null > "${adb_tmpfile}"
             if [ -s "${adb_tmpfile}" ]
             then
-                if [ ${adb_tldcomp} -eq 1 ]
-                then
-                    awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" | sort -u > "${adb_tmpload}"
-                    awk '{if(NR==1){tld=$NF};while(getline){if($NF !~ tld"\\."){print tld;tld=$NF}}print tld}' "${adb_tmpload}" > "${adb_tmpfile}"
-                    awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" > "${adb_tmpload}"
-                else
-                    sort -u "${adb_tmpfile}" > "${adb_tmpload}"
-                fi
+                awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" 2>/dev/null | sort -u > "${adb_tmpload}"
+                awk '{if(NR==1){tld=$NF};while(getline){if($NF !~ tld"\\."){print tld;tld=$NF}}print tld}' "${adb_tmpload}" 2>/dev/null > "${adb_tmpfile}"
+                awk -F "." '{for(f=NF;f > 1;f--) printf "%s.", $f;print $1}' "${adb_tmpfile}" 2>/dev/null > "${adb_tmpload}"
                 mv -f "${adb_tmpload}" "${adb_tmpfile}"
                 f_list backup
             else
@@ -424,9 +417,9 @@ f_main()
         then
             if [ -s "${adb_tmpdir}/tmp.whitelist" ]
             then
-                grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}"
+                grep -vf "${adb_tmpdir}/tmp.whitelist" "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" > "${adb_dnsfile}"
             else
-                cat "${adb_tmpfile}" | eval "${adb_dnsformat}" > "${adb_dnsfile}"
+                cat "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" > "${adb_dnsfile}"
             fi
             adb_rc=${?}
             if [ ${adb_rc} -ne 0 ]
@@ -470,18 +463,24 @@ f_main()
     f_dnsrestart
     if [ "${adb_dnsup}" = "true" ]
     then
-        f_log "info " "block lists with overall ${sum_cnt} domains loaded successfully (${sysver})"
-        ubus call service set "{\"name\":\"adblock_stats\",
-            \"instances\":{\"statistics\":{\"command\":[\"\"],
+        f_log "info " "block lists with overall ${sum_cnt} domains loaded successfully (${adb_sysver})"
+        for name in ${adb_iface}
+        do
+            active_triggers="${active_triggers}[\"interface.*.up\",[\"if\",[\"eq\",\"interface\",\"${name}\"],[\"run_script\",\"/etc/init.d/adblock\",\"start\"],1000]],"
+        done
+        active_triggers="${active_triggers}[\"config.change\",[\"if\",[\"eq\",\"package\",\"adblock\"],[\"run_script\",\"/etc/init.d/adblock\",\"start\"],1000]]"
+        ubus call service set "{\"name\":\"adblock\",
+            \"instances\":{\"adblock\":{\"command\":[\"/usr/bin/adblock.sh\"],
             \"data\":{\"active_lists\":[{${active_lists}}],
             \"adblock_version\":\"${adb_ver}\",
             \"blocked_domains\":\"${sum_cnt}\",
             \"dns_backend\":\"${adb_dns}\",
             \"last_rundate\":\"$(/bin/date "+%d.%m.%Y %H:%M:%S")\",
-            \"system\":\"${sysver}\"}}}}"
-        return 0
+            \"system\":\"${adb_sysver}\"}}},
+            \"triggers\":[${active_triggers}]}"
+    else
+        f_log "error" "dns backend restart with active block lists failed"
     fi
-    f_log "error" "dns backend restart with active block lists failed (${sysver})"
 }
 
 # handle different adblock actions