26 KiB

Raw Blame History

DNS based ad/abuse domain blocking

Description

A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other (wlan) gadget!? Getting rid of annoying ads, trackers and other abuse sites (like facebook) is simple: block them with your router. When the DNS server on your router receives DNS requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but Non-eXistent Internet or Intranet domain name, if domain name is unable to resolved using the DNS server, a condition called the 'NXDOMAIN' occurred.

Main Features

Support of the following fully pre-configured domain blocklist sources (free for private usage, for commercial use please check their individual licenses)

Source	Enabled	Size	Focus	Information
adaway	x	S	mobile	Link
adguard	x	L	general	Link
anti_ad		L	compilation	Link
andryou		L	compilation	Link
anudeep		M	compilation	Link
bitcoin		S	mining	Link
disconnect	x	S	general	Link
dshield		XL	general	Link
energized_blugo		XL	compilation	Link
energized_blu		XL	compilation	Link
energized_porn		XXL	compilation+porn	Link
energized_ultimate		XXL	compilation	Link
energized_unified		XXL	compilation	Link
malwaredomains		M	malware	Link
malwarelist		S	malware	Link
notracking		XL	tracking	Link
oisd_nl		XXL	general	Link
openphish		S	phishing	Link
phishing_army		S	phishing	Link
reg_cn		M	reg_china	Link
reg_cz		M	reg_czech+slovak	Link
reg_de		M	reg_germany	Link
reg_es		M	reg_espania	Link
reg_fi		S	reg_finland	Link
reg_fr		S	reg_france	Link
reg_id		M	reg_indonesia	Link
reg_kr		M	reg_korea	Link
reg_nl		M	reg_netherlands	Link
reg_pl		M	reg_poland	Link
reg_ro		M	reg_romania	Link
reg_ru		M	reg_russia	Link
reg_vn		M	reg_vietnam	Link
shallalist		L	general	Link
shallalist_porn		XXL	general+porn	Link
smarttv		S	smarttv	Link
spam404		S	general	Link
stevenblack		L	compilation	Link
stevenblack_porn		L	compilation+porn	Link
stopforumspam		S	spam	Link
sysctl		M	general	Link
utcapitole		L	general	Link
utcapitole_porn		XXL	general+porn	Link
wally3k		S	compilation	Link
whocares		M	general	Link
winhelp		S	general	Link
winspy		S	win_telemetry	Link
youtube		M	youtube	Link
yoyo	x	S	general	Link

List of supported and fully pre-configured adblock sources, already active sources are pre-selected.
To avoid OOM errors, please do not select too many lists!
List size information with the respective domain ranges as follows:
• S (-10k), M (10k-30k) and L (30k-80k) should work for 128 MByte devices,
• XL (80k-200k) should work for 256-512 MByte devices,
• XXL (200k-) needs more RAM and Multicore support, e.g. x86 or raspberry devices.
Zero-conf like automatic installation & setup, usually no manual changes needed
Simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
Supports five different DNS backend formats: dnsmasq, unbound, named (bind), kresd or raw (e.g. used by dnscrypt-proxy)
Supports four different SSL-enabled download utilities: uclient-fetch, wget, curl or aria2c
Supports SafeSearch for google, bing, duckduckgo, yandex, youtube and pixabay
Fast downloads & list processing as they are handled in parallel running background jobs with multicore support
Supports a wide range of router modes, even AP modes are supported
Full IPv4 and IPv6 support
Provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the DNS backend
Provides a 'DNS File Reset', where the generated DNS blocklist file will be purged after DNS backend loading to save storage space
Source parsing by fast & flexible regex rulesets, all rules and source information are placed in an external/compredd JSON file ('/etc/adblock/adblock.sources.gz')
Overall duplicate removal in generated blocklist file 'adb_list.overall'
Additional local blacklist for manual overrides, located in '/etc/adblock/adblock.blacklist'
Additional local whitelist for manual overrides, located in '/etc/adblock/adblock.whitelist'
Quality checks during blocklist update to ensure a reliable DNS backend service
Minimal status & error logging to syslog, enable debug logging to receive more output
Procd based init system support ('start', 'stop', 'restart', 'reload', 'enable', 'disable', 'running', 'status', 'suspend', 'resume', 'query', 'report', 'list', 'timer')
Auto-Startup via procd network interface trigger or via classic time based startup
Suspend & Resume adblock temporarily without blocklist reloading
Provides comprehensive runtime information
Provides a detailed DNS Query Report with DNS related information about client requests, top (blocked) domains and more
Provides a powerful query function to quickly find blocked (sub-)domains, e.g. for whitelisting
Provides an easily configurable blocklist update scheduler called 'Refresh Timer'
Includes an option to generate an additional, restrictive 'adb_list.jail' to block access to all domains except those listed in the whitelist file. You can use this restrictive blocklist manually e.g. for guest wifi or kidsafe configurations
Includes an option to force DNS requests to the local resolver
Automatic blocklist backup & restore, these backups will be used in case of download errors and during startup
Send notification E-Mails in case of a processing error or if the overall domain count is ≤ 0
Add new adblock sources on your own, see example below
Strong LuCI support, all relevant options are exposed to the web frontend

Prerequisites

OpenWrt, tested with the stable release series (19.07.x) and with the latest rolling snapshot releases. On turris devices it has been successfully tested with TurrisOS 5.1.x
Please note: Older OpenWrt releases like 18.06.x or 17.01.x are not supported!
Please note: Devices with less than 128 MByte RAM are not supported!
A usual setup with an enabled DNS backend at minimum - dump AP modes without a working DNS backend are not supported
A download utility with SSL support: 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
A certificate store such as 'ca-bundle' or 'ca-certificates', as adblock checks the validity of the SSL certificates of all download sites by default
Optional E-Mail notification support: for E-Mail notifications you need to install the additional 'msmtp' package
Optional DNS Query Report support: for DNS reporting you need to install the additional package 'tcpdump-mini' or 'tcpdump'
Optional support for gnu awk as alternative to the busybox default, install the additional package 'gawk'

Installation & Usage

Update your local opkg repository (opkg update)
Install 'adblock' (opkg install adblock). The adblock service is enabled by default
Install the LuCI companion package 'luci-app-adblock' (opkg install luci-app-adblock)
It's strongly recommended to use the LuCI frontend to easily configure all aspects of adblock, the application is located in LuCI under the 'Services' menu
Update from a former adblock version is easy. During the update a backup is made of the old configuration '/etc/config/adblock-backup' and replaced by the new config - that's all

Adblock CLI Options

All important adblock functions are accessible via CLI as well.
Please note: The 'status' command in 19.07.x and TurrisOS is only available via 'status_service'


/etc/init.d/adblock 
Syntax: /etc/init.d/adblock [command]

Available commands:
	start	Start the service
	stop	Stop the service
	restart	Restart the service
	reload	Reload configuration files (or restart if service does not implement reload)
	enable	Enable service autostart
	disable	Disable service autostart
	running	Check if service is running
	status	Service status
	suspend	Suspend adblock processing
	resume	Resume adblock processing
	query	<domain> Query active blocklists and backups for a specific domain
	report	[<search>] Print DNS statistics with an optional search parameter
	list	[[<add>|<remove>] [source(s)]] List available adblock sources or add/remove them from config
	timer	<action> <hour> [<minute>] [<weekday>] Set a cron based update interval

Adblock Config Options

Usually the auto pre-configured adblock setup works quite well and no manual overrides are needed

Option	Default	Description/Valid Values
adb_enabled	1, enabled	set to 0 to disable the adblock service
adb_srcarc	-, /etc/adblock/adblock.sources.gz	full path to the used adblock source archive
adb_srcfile	-, /tmp/adb_sources.json	full path to the used adblock source file, which has a higher precedence than the archive file
adb_dns	-, auto-detected	'dnsmasq', 'unbound', 'named', 'kresd' or 'raw'
adb_fetchutil	-, auto-detected	'uclient-fetch', 'wget', 'curl' or 'aria2c'
adb_fetchparm	-, auto-detected	config options for the selected download utility, e.g. to disable the certificate check
adb_trigger	-, not set	trigger network interface or 'not set' to use a time-based startup
adb_triggerdelay	2	additional trigger delay in seconds before adblock processing begins
adb_debug	0, disabled	set to 1 to enable the debug output
adb_nice	0, standard prio.	valid nice level range 0-19 of the adblock processes
adb_forcedns	0, disabled	set to 1 to force DNS requests to the local resolver
adb_maxqueue	4	size of the download queue to handle downloads & list processing in parallel
adb_dnsdir	-, auto-detected	path for the generated blocklist file 'adb_list.overall'
adb_dnstimeout	10	timeout in seconds to wait for a successful DNS backend restart
adb_dnsinstance	0, first instance	set to the relevant dns backend instance used by adblock (dnsmasq only)
adb_dnsfilereset	0, disabled	set to 1 to purge the final DNS blocklist file after DNS backend loading
adb_dnsflush	0, disabled	set to 1 to flush the DNS Cache before & after adblock processing
adb_dnsinotify	-, not set	set to 1 to prevent adblock triggered restarts for DNS backends with autoload functions
adb_dnsallow	-, not set	set to 1 to disable selective DNS whitelisting (RPZ pass through)
adb_lookupdomain	example.com	external domain to check for a successful DNS backend restart or 'false' to disable this check
adb_portlist	53 853 5353	space separated list of firewall ports which should be redirected locally
adb_report	0, disabled	set to 1 to enable the background tcpdump gathering process for reporting
adb_reportdir	/tmp	path for DNS related report files
adb_repiface	-, auto-detected	name of the reporting interface or 'any' used by tcpdump
adb_replisten	53	space separated list of reporting port(s) used by tcpdump
adb_repchunkcnt	5	report chunk count used by tcpdump
adb_repchunksize	1	report chunk size used by tcpdump in MB
adb_backup	1, enabled	set to 0 to disable the backup function
adb_backupdir	/tmp	path for adblock backups
adb_tmpbase	/tmp	path for all adblock related runtime operations, e.g. downloading, sorting, merging etc.
adb_safesearch	0, disabled	set to 1 to enforce SafeSearch for google, bing, duckduckgo, yandex, youtube and pixabay
adb_safesearchlist	-, not set	Limit SafeSearch to certain provider (see above)
adb_safesearchmod	0, disabled	set to 1 to enable moderate SafeSearch filters for youtube
adb_mail	0, disabled	set to 1 to enable notification E-Mails in case of a processing errors
adb_mailreceiver	-, not set	receiver address for adblock notification E-Mails
adb_mailsender	no-reply@adblock	sender address for adblock notification E-Mails
adb_mailtopic	adblock notification	topic for adblock notification E-Mails
adb_mailprofile	adb_notify	mail profile used in 'msmtp' for adblock notification E-Mails
adb_mailcnt	0	minimum domain count to trigger E-Mail notifications
adb_jail	0	set to 1 to enable the additional, restrictive 'adb_list.jail' creation
adb_jaildir	/tmp	path for the generated jail list

Examples

Change the DNS backend to 'unbound':
No further configuration is needed, adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/unbound' by default.

Change the DNS backend to 'named' (bind):
Adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/bind'.
To preserve the DNS cache after adblock processing you need to install & configure 'bind-rdnc'.
To use the blocklist please modify '/etc/bind/named.conf':


in the 'options' namespace add:
  response-policy { zone "rpz"; };

and at the end of the file add:
  zone "rpz" {
    type master;
    file "/var/lib/bind/adb_list.overall";
    allow-query { none; };
    allow-transfer { none; };
  };

Change the DNS backend to 'kresd':
Adblock deposits the final blocklist 'adb_list.overall' in '/etc/kresd', no further configuration needed.
Please note: The knot-resolver (kresd) is only available on Turris devices and does not support the SafeSearch functionality yet.

Enable E-Mail notification via 'msmtp':
To use the email notification you have to install & configure the package 'msmtp'.
Modify the file '/etc/msmtprc':


[...]
defaults
auth            on
tls             on
tls_certcheck   off
timeout         5
syslog          LOG_MAIL
[...]
account         adb_notify
host            smtp.gmail.com
port            587
from            dev.adblock@gmail.com
user            dev.adblock
password        xxx

Finally enable E-Mail support and add a valid E-Mail receiver address in LuCI.

Service status output:
In LuCI you'll see the realtime status in the 'Runtime' section on the overview page.
To get the status in the CLI, just call /etc/init.d/adblock status or /etc/init.d/adblock status_service (in 19.07 and TurrisOS):


/etc/init.d/adblock status_service
::: adblock runtime information
  + adblock_status  : enabled
  + adblock_version : 4.0.2
  + blocked_domains : 52420
  + active_sources  : adaway adguard andryou bitcoin disconnect winspy yoyo 
  + dns_backend     : kresd, /etc/kresd
  + run_utils       : /usr/bin/curl, /bin/awk
  + run_ifaces      : trigger: trm_wwan, report: br-lan
  + run_directories : base: /tmp, backup: /tmp, report: /tmp, jail: /tmp
  + run_flags       : backup: 1, reset: 0, flush: 0, force: 1, search: 0, report: 1, mail: 0, jail: 0
  + last_run        : start, 0m 17s, 496/198/218, 03.04.2020 08:55:14
  + system          : CZ.NIC Turris Mox Board, TurrisOS 5.1.0 81264ebb51991aa2d17489852854e3b5ec3f514d

The 'last_run' line includes the used start type, the run duration, the memory footprint after DNS backend loading (total/free/available) and the date/time of the last run.

Edit, add new adblock sources:
The adblock blocklist sources are stored in an external, compressed JSON file '/etc/adblock/adblock.sources.gz'. This file is directly parsed in LuCI and accessible via CLI, just call /etc/init.d/adblock list:


/etc/init.d/adblock list
::: Available adblock sources
:::
    Name                 Enabled   Size   Focus               Info URL
    ------------------------------------------------------------------
  + adaway               x         S      mobile              https://adaway.org
  + adguard              x         L      general             https://adguard.com
  + andryou              x         L      compilation         https://gitlab.com/andryou/block/-/blob/master/readme.md
  + bitcoin              x         S      mining              https://github.com/hoshsadiq/adblock-nocoin-list
  + disconnect           x         S      general             https://disconnect.me
  + dshield                        XL     general             https://www.dshield.org
[...]
  + winhelp                        S      general             http://winhelp2002.mvps.org
  + winspy               x         S      win_telemetry       https://github.com/crazy-max/WindowsSpyBlocker
  + yoyo                 x         S      general             https://pgl.yoyo.org

To add new or edit existing sources extract the compressed JSON file gunzip /etc/adblock/adblock.sources.gz.
A valid JSON source object contains the following required information, e.g.:


	[...]
	"adaway": {
		"url": "https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt",
		"rule": "/^127\\.0\\.0\\.1[[:space:]]+([[:alnum:]_-]+\\.)+[[:alpha:]]+([[:space:]]|$)/{print tolower($2)}",
		"size": "S",
		"focus": "mobile",
		"descurl": "https://github.com/AdAway/adaway.github.io"
	},
	[...]

Add an unique object name, make the required changes to 'url', 'rule', 'size' and 'descurl' and finally compress the changed JSON file gzip /etc/adblock/adblock.sources.gz to use the new source object in adblock.
Please note: if you're going to add new sources on your own, please make a copy of the default file and work with that copy further on, cause the default will be overwritten with every adblock update. To reference your copy set the option 'adb_srcarc' which points by default to '/etc/adblock/adblock.sources.gz'
Please note: when adblock starts, it looks for the uncompressed 'adb_srcfile', only if this file is not found the archive 'adb_srcarc' is unpacked once and then the uncompressed file is used

Support

Please join the adblock discussion in this forum thread or contact me by mail dev@brenken.org

Have fun!
Dirk

26 KiB Raw Blame History