* major performance boost: add a flexible 'Download Queue' to handle
downloads & list processing in parallel, default queue size is '4',
you can raise this e.g. to '8' or '16' to get it really fast
* replace former 'whitelist mode': the new 'Jail' option
builds an additional 'adb_list.jail' list in parallel
which can be used manually for guest wifi or kidsafe configurations
* regex parser & query function now fully support IDN domains
with non-ASCII characters
* add error handling in tld compression,
to handle OOM conditions better
* adblock.notify sends now html emails,
to get a better look & feel, even on mobile devices
* add czech regional blocklist maintained by turris omnia users
* LuCI: Support new 'Download Queue' & 'Jail' options
* LuCI: fix field width in "Runtime Information" section
Signed-off-by: Dirk Brenken <dev@brenken.org>
@ -63,10 +65,11 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
* simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
* supports five different dns backends / blocklist formats: dnsmasq, unbound, named (bind), kresd and dnscrypt-proxy
* supports five different dns backends / blocklist formats: dnsmasq, unbound, named (bind), kresd and dnscrypt-proxy
* supports six different download utilities: uclient-fetch, wget, curl, aria2c, wget-nossl, busybox-wget
* supports six different download utilities: uclient-fetch, wget, curl, aria2c, wget-nossl, busybox-wget
* Really fast downloads & list processing as they are handled in parallel as background jobs in a configurable 'Download Queue'
* provides 'http only' mode without installed ssl library for all non-SSL blocklist sources
* provides 'http only' mode without installed ssl library for all non-SSL blocklist sources
* supports a wide range of router modes, even AP modes are supported
* supports a wide range of router modes, even AP modes are supported
* full IPv4 and IPv6 support
* full IPv4 and IPv6 support
* provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the dns backends
* provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the dns backend
* blocklist source parsing by fast & flexible regex rulesets
* blocklist source parsing by fast & flexible regex rulesets
* overall duplicate removal in central blocklist 'adb_list.overall'
* overall duplicate removal in central blocklist 'adb_list.overall'
* additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
* additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
@ -74,7 +77,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* minimal status & error logging to syslog, enable debug logging to receive more output
* minimal status & error logging to syslog, enable debug logging to receive more output
* procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
* procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
* procd network interface trigger support or classic time based startup
* procd network interface trigger support or classic time based startup
* keep the dns cache intact after adblock processing (currently supported by unbound and named)
* keep the dns cache intact after adblock processing (currently supported by unbound, named and kresd)
* conditional dns backend restarts by old/new blocklist comparison with sha256sum (default) or md5sum
* conditional dns backend restarts by old/new blocklist comparison with sha256sum (default) or md5sum
* suspend & resume adblock actions temporarily without blocklist reloading
* suspend & resume adblock actions temporarily without blocklist reloading
* output comprehensive runtime information via LuCI or via 'status' init command
* output comprehensive runtime information via LuCI or via 'status' init command
@ -84,12 +87,12 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* optional: force overall sort / duplicate removal for low memory devices (handle with care!)
* optional: force overall sort / duplicate removal for low memory devices (handle with care!)
* optional: automatic blocklist backup & restore, they will be used in case of download errors or during startup in backup mode
* optional: automatic blocklist backup & restore, they will be used in case of download errors or during startup in backup mode
* optional: 'backup mode' to re-use blocklist backups during startup, get fresh lists only via reload or restart action
* optional: 'backup mode' to re-use blocklist backups during startup, get fresh lists only via reload or restart action
* optional: 'whitelist mode' to block access to all domains except those explicitly listed in the whitelist file
* optional: 'Jail' blocklist generation which builds an additional list (/tmp/adb_list.jail) to block access to all domains except those listed in the whitelist file. You can use this restrictive blocklist manually e.g. for guest wifi or kidsafe configurations
* optional: send notification emails in case of a processing error or if the overall domain count is ≤ 0
* optional: send notification emails in case of a processing error or if the overall domain count is ≤ 0
* optional: add new adblock sources on your own via uci config
* optional: add new adblock sources on your own, see example below
## Prerequisites
## Prerequisites
* [LEDE project](https://www.lede-project.org), tested with latest stable release (LEDE 17.01) and with current LEDE snapshot
* [OpenWrt](https://openwrt.org), tested with the stable release series (17.01.x) and with the latest OpenWrt snapshot
* a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
* a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
* a download utility:
* a download utility:
* to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
* to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
@ -98,7 +101,7 @@ A lot of people already use adblocker plugins within their desktop browsers, but
## Installation & Usage
## Installation & Usage
* install 'adblock' (_opkg install adblock_)
* install 'adblock' (_opkg install adblock_)
* at minimum configure the appropriate dns backend ('dnsmasq' by default) and enable the adblock service in _/etc/config/adblock_
* at minimum configure the appropriate dns backend ('dnsmasq' by default), the donwload utility and enable the adblock service in _/etc/config/adblock_
* control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
* control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
## LuCI adblock companion package
## LuCI adblock companion package
@ -109,13 +112,14 @@ A lot of people already use adblocker plugins within their desktop browsers, but
## Tweaks
## Tweaks
* **runtime information:** the adblock status is available via _/etc/init.d/adblock status_ (see example below)
* **runtime information:** the adblock status is available via _/etc/init.d/adblock status_ (see example below)
* **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
* **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
* **storage expansion:** to process and store all blocklist sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
* **storage expansion:** to process and store all blocklist sources at once it might helpful to enlarge your temp directory with a swap partition => see [OpenWrt Wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
* **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
* **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
* **backup & restore blocklists:** enable this feature, to restore automatically the latest compressed backup of your blocklists in case of any processing error (e.g. a single blocklist source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
* **backup & restore blocklists:** enable this feature, to restore automatically the latest compressed backup of your blocklists in case of any processing error (e.g. a single blocklist source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
* **download queue size:** for further download & list processing performance improvements you can raise the 'adb\_maxqueue' value, e.g. '8' or '16' should be safe
* **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
* **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
* **change startup behaviour:** by default the startup will be triggered by the 'wan' procd interface trigger. Choose 'none' to disable automatic startups, 'timed' to use a classic timeout (default 30 sec.) or select another trigger interface.
* **change startup behaviour:** by default the startup will be triggered by the 'wan' procd interface trigger. Choose 'none' to disable automatic startups, 'timed' to use a classic timeout (default 30 sec.) or select another trigger interface
* **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
* **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
* **domain query:** to query the active blocklist for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
* **domain query:** to query the active blocklist for a certain domain, please use the LuCI frontend or run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
* **add new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
* **add new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
* **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
* **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
@ -131,14 +135,15 @@ A lot of people already use adblocker plugins within their desktop browsers, but
* adb\_trigger => set the startup trigger to a certain interface, to 'timed' or to 'none' (default: 'wan')
* adb\_trigger => set the startup trigger to a certain interface, to 'timed' or to 'none' (default: 'wan')
* the following options apply to the 'extra' config section:
* the following options apply to the 'extra' config section:
* adb\_triggerdelay => additional trigger delay in seconds before adblock processing begins (default: '1')
* adb\_forcedns => force dns requests to local resolver (default: '0', disabled)
* adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (default: '0', disabled)
* adb\_backup_mode => do not automatically update blocklists during startup, use backups instead (default: '0', disabled)
* adb\_whitelist_mode => block access to all domains except those explicitly listed in the whitelist file (default: '0', disabled)
* adb\_dnsflush => flush DNS cache after adblock processing, i.e. enable the old restart behavior (default: '0', disabled)
* adb\_notify => send notification emails in case of a processing error or if the overall domain count is ≤ 0 (default: '0', disabled)
* adb\_triggerdelay => additional trigger delay in seconds before adblock processing begins (int/default: '2')
* adb\_forcedns => force dns requests to local resolver (bool/default: '0', disabled)
* adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (bool/default: '0', disabled)
* adb\_backup_mode => do not automatically update blocklists during startup, use backups instead (bool/default: '0', disabled)
* adb\maxqueue => size of the download queue to handle downloads & list processing in parallel (int/default: '4')
* adb\_jail => builds an additional 'Jail' list (/tmp/adb_list.jail) to block access to all domains except those listed in the whitelist file (bool/default: '0', disabled)
* adb\_dnsflush => flush DNS cache after adblock processing, i.e. enable the old restart behavior (bool/default: '0', disabled)
* adb\_notify => send notification emails in case of a processing error or if the overall domain count is ≤ 0 (bool/default: '0', disabled)
@ -172,7 +177,7 @@ Adblock deposits the final blocklist 'adb_list.overall' in '/etc/kresd', no furt
**change default dns backend to 'dnscrypt-proxy':**
**change default dns backend to 'dnscrypt-proxy':**
The required 'blacklist' option of dnscrypt-proxy is not enabled by default, because the package will be compiled without plugins support.
The required 'blacklist' option of dnscrypt-proxy is not enabled by default, because the package will be compiled without plugins support.
Take a custom LEDE build with plugins support to use this feature. Adblock deposits the final blocklist 'adb_list.overall' in '/tmp'.
Take a custom OpenWrt build with plugins support to use this feature. Adblock deposits the final blocklist 'adb_list.overall' in '/tmp'.
To use the blocklist please modify '/etc/config/dnscrypt-proxy' per instance:
To use the blocklist please modify '/etc/config/dnscrypt-proxy' per instance:
<pre><code>
<pre><code>
list blacklist 'domains:/tmp/adb_list.overall'
list blacklist 'domains:/tmp/adb_list.overall'
@ -207,12 +212,12 @@ Finally make this file executable via 'chmod' and test it directly. If no more e
/etc/init.d/adblock status
/etc/init.d/adblock status
::: adblock runtime information
::: adblock runtime information
+ adblock_status : enabled
+ adblock_status : enabled
+ adblock_version : 3.4.0
+ overall_domains : 5167 (normal/backup mode)
+ fetch_utility : wget (built-in)
+ adblock_version : 3.5.0
+ overall_domains : 102324 (normal mode)
+ fetch_utility : /usr/bin/wget (built-in)
+ dns_backend : kresd (/etc/kresd)
+ dns_backend : kresd (/etc/kresd)
+ last_rundate : 27.12.2017 20:52:35
+ system_release : Turris Omnia, OpenWrt omnia 15.05/3.9.1
+ last_rundate : 30.01.2018 21:24:11
+ system_release : Turris Omnia, OpenWrt omnia 15.05/3.9.4
</code></pre>
</code></pre>
**cronjob for a regular block list update (/etc/crontabs/root):**
**cronjob for a regular block list update (/etc/crontabs/root):**
@ -255,18 +260,22 @@ This entry does not remove:
The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain. For every (sub-)domain it returns the first ten relevant results.
The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain. For every (sub-)domain it returns the first ten relevant results.
<pre><code>
<pre><code>
/etc/init.d/adblock query www.example.google.com
/etc/init.d/adblock query www.example.google.com
::: max. ten results for domain 'www.example.google.com'
::: results for domain 'www.example.google.com'
- no match
- no match
::: max. ten results for domain 'example.google.com'
::: results for domain 'example.google.com'
- no match
- no match
::: max. ten results for domain 'google.com'
::: results for domain 'google.com'
+ ads.google.com
+ adservices.google.com
+ adwords.google.com
+ ampcid.google.com
+ analytics.google.com
+ analytics.google.com
+ gg.google.com
+ google.com.analytics.kdgsrltkcun.com
+ googleadapis.l.google.com
+ googleadapis.l.google.com
+ pagead.l.google.com
+ partnerad.l.google.com
+ ssl-google-analytics.l.google.com
+ video-stats.video.google.com
+ www-google-analytics.l.google.com
+ id.google.com
+ pagead-googlehosted.l.google.com
+ [...]
</code></pre>
</code></pre>
**add a new blocklist source:**
**add a new blocklist source:**
@ -279,7 +288,7 @@ the source name, the url and the description - that's all!
Dirk Brenken
7 years ago