You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

368 lines
22 KiB

  1. # DNS based ad/abuse domain blocking
  2. ## Description
  3. A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other (wlan) gadget!? Getting rid of annoying ads, trackers and other abuse sites (like facebook) is simple: block them with your router. When the DNS server on your router receives DNS requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the DNS server, a condition called the 'NXDOMAIN' occurred.
  4. ## Main Features
  5. * Support of the following domain blocklist sources (free for private usage, for commercial use please check their individual licenses):
  6. * [adaway](https://adaway.org)
  7. * Infrequent updates, approx. 400 entries (enabled by default)
  8. * [adguard](https://adguard.com)
  9. * Numerous updates on the same day, approx. 12.000 entries
  10. * [bitcoin](https://github.com/hoshsadiq/adblock-nocoin-list)
  11. * Infrequent updates, approx. 15 entries
  12. * [blacklist]()
  13. * Static local blacklist, located by default in `/etc/adblock/adblock.blacklist`
  14. * [disconnect](https://disconnect.me)
  15. * Numerous updates on the same day, approx. 6.500 entries (enabled by default)
  16. * [dshield](http://dshield.org)
  17. * Daily updates, approx. 4.500 entries
  18. * [hphosts](https://hosts-file.net)
  19. * Monthly updates, approx. 50.000 entries
  20. * [malwaredomains](http://malwaredomains.com)
  21. * Daily updates, approx. 16.000 entries
  22. * [malwaredomainlist](http://www.malwaredomainlist.com)
  23. * Daily updates, approx. 1.500 entries
  24. * [openphish](https://openphish.com)
  25. * Numerous updates on the same day, approx. 1.800 entries
  26. * [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt)
  27. * Regional blocklist for China, daily updates, approx. 1.600 entries
  28. * [reg_cz](https://raw.githubusercontent.com/qxstyles/turris-hole-czech-block-list/master/turris-hole-czech-block-list)
  29. * Regional blocklist for Czechia, maintained by Turris Omnia Users, infrequent updates, approx. 100 entries
  30. * [reg_de](https://easylist-downloads.adblockplus.org/easylistgermany+easylist.txt)
  31. * Regional blocklist for Germany, daily updates, approx. 9.200 entries
  32. * [reg_id](https://easylist-downloads.adblockplus.org/abpindo+easylist.txt)
  33. * Regional blocklist for Indonesia, daily updates, approx. 800 entries
  34. * [reg_nl](https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt)
  35. * Regional blocklist for the Netherlands, weekly updates, approx. 1300 entries
  36. * [reg_pl](http://adblocklist.org)
  37. * Regional blocklist for Poland, daily updates, approx. 50 entries
  38. * [reg_ro](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  39. * Regional blocklist for Romania, weekly updates, approx. 600 entries
  40. * [reg_ru](https://code.google.com/p/ruadlist)
  41. * Regional blocklist for Russia, weekly updates, approx. 2.000 entries
  42. * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  43. * Daily updates, approx. 32.000 entries (a short description of all categories can be found [online](http://www.shallalist.de/categories.html))
  44. * [spam404](http://www.spam404.com)
  45. * Infrequent updates, approx. 5.000 entries
  46. * [sysctl/cameleon](http://sysctl.org/cameleon)
  47. * Weekly updates, approx. 21.000 entries
  48. * [ut_capitole](https://dsi.ut-capitole.fr/blacklists) (categories "cryptojacking" "ddos" "malware" "phishing" "warez" enabled by default)
  49. * Daily updates, approx. 64.000 entries (a short description of all categories can be found [online](https://dsi.ut-capitole.fr/blacklists/index_en.php))
  50. * [whocares](https://someonewhocares.org)
  51. * Weekly updates, approx. 12.000 entries
  52. * [winhelp](http://winhelp2002.mvps.org)
  53. * Infrequent updates, approx. 15.000 entries
  54. * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  55. * Infrequent updates, approx. 120 entries
  56. * [yoyo](http://pgl.yoyo.org/adservers)
  57. * Weekly updates, approx. 2.500 entries (enabled by default)
  58. * Zero-conf like automatic installation & setup, usually no manual changes needed
  59. * Simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  60. * Support four different DNS backends: `dnsmasq`, `unbound`, `named` (bind) and `kresd`
  61. * Support two different DNS blocking variants: `nxdomain` (default, supported by all backends), `null` (supported only by `dnsmasq`)
  62. * Support six different download utilities: `uclient-fetch`, `wget`, `curl`, `aria2c`, `wget-nossl`, `busybox-wget`
  63. * Fast downloads & list processing as they are handled in parallel running background jobs (see 'Download Queue')
  64. * Provide `http only` mode without installed SSL library for all non-SSL blocklist sources
  65. * Support a wide range of router modes, even AP modes are supported
  66. * Full IPv4 and IPv6 support
  67. * Provide top level domain compression (`tld compression`), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the DNS backend
  68. * Provide a 'DNS File Reset', where the final DNS blockfile will be purged after DNS backend loading to save storage space
  69. * Blocklist source parsing by fast & flexible regex rulesets
  70. * Overall duplicate removal in central blocklist `adb_list.overall`
  71. * Additional blacklist for manual overrides, located by default in `/etc/adblock/adblock.blacklist` or in LuCI
  72. * Additional whitelist for manual overrides, located by default in `/etc/adblock/adblock.whitelist` or in LuCI
  73. * Quality checks during blocklist update to ensure a reliable DNS backend service
  74. * Minimal status & error logging to syslog, enable debug logging to receive more output
  75. * procd based init system support (`start/stop/restart/reload/suspend/resume/query/status`)
  76. * procd network interface trigger support or classic time based startup
  77. * Keep the DNS cache intact after adblock processing (currently supported by unbound, named and kresd)
  78. * Suspend & resume adblock actions temporarily without blocklist reloading
  79. * Provide comprehensive runtime information via LuCI or via `status` init command
  80. * Provide a detailed DNS Query Report with DNS related information about client requests, top (blocked) domains and more
  81. * Provide a query function to quickly identify blocked (sub-)domains, e.g. for whitelisting. This function is also able to search in adblock backups and black-/whitelist, to get back the set of blocking lists sources for a certain domain
  82. * Option to force DNS requests to the local resolver
  83. * Automatic blocklist backup & restore, these backups will be used in case of download errors and during startup
  84. * Send notification emails in case of a processing error or if the overall domain count is ≤ 0
  85. * Add new adblock sources on your own, see example below
  86. * Strong LuCI support for all options
  87. ## Installation & Usage
  88. ### Prerequisites
  89. * [OpenWrt](https://openwrt.org), tested with the stable release series (19.07) and with the latest snapshot
  90. * A usual setup with an enabled DNS backend at minimum - dump AP modes without a working DNS backend are _not_ supported
  91. * A download utility:
  92. * To support all blocklist sources and in order to run the default configuration of `adblock`, a full version (with SSL support) of `wget`, `uclient-fetch` with one of the `libustream-*` SSL libraries, `aria2c` or `curl` is required
  93. * The package used by default is probably `uclient-fetch` so in order to make `adblock` work with its default configuration it is needed to install one of the `libustream-*` SSL libraries. Example: `opkg install libustream-openssl`
  94. * For limited devices with real memory constraints, adblock provides also a `http only` option and supports `wget-nossl` and `uclient-fetch` (without `libustream-ssl`) as well
  95. * For more configuration options see examples below
  96. * Email notification (optional): For email notification support you need the additional `msmtp` package
  97. * DNS Query Report (optional): For this detailed report you need the additional package `tcpdump` or `tcpdump-mini`
  98. ### Installation of the core package
  99. * Install `adblock` (`opkg install adblock`)
  100. ### LuCI adblock companion package
  101. * It is strongly recommended to use the LuCI frontend to easily configure all powerful aspects of adblock
  102. * Install `luci-app-adblock` (`opkg install luci-app-adblock`)
  103. * The application is located in LuCI under the `Services` menu
  104. ### Configuration and controlling
  105. * At minimum configure the appropriate DNS backend (`dnsmasq` by default), the download utility and enable the adblock service in `/etc/config/adblock`
  106. * Control the adblock service manually with `/etc/init.d/adblock` `start/stop/restart/reload/suspend/resume/status` or use the LuCI frontend
  107. #### Tweaks
  108. * **Runtime information:** The adblock status is available via `/etc/init.d/adblock status` (see example below)
  109. * **Debug logging:** For script debugging please set the config option `adb\_debug` to `1` and check the runtime output with `logread -e "adblock"`
  110. * **Storage expansion:** To process and store all blocklist sources at once it might be helpful to enlarge your temp directory with a swap partition => see [OpenWrt Wiki](https://openwrt.org/docs/guide-user/storage/fstab) for further details
  111. * **coreutils sort:** To speedup adblock processing in particular with many enabled blocklist sources it is recommended to install the additional package `coreutils-sort`
  112. * **Add white- / blacklist entries:** Add domain black- or whitelist entries to always-deny or -allow certain (sub) domains, by default both lists are empty and located in `/etc/adblock`. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below). You need to refresh your blocklists after changes to these static lists.
  113. * **Download queue size:** For further download & list processing performance improvements you can raise the `adb\_maxqueue` value, e.g. `8` or `16` should be safe
  114. * **Scheduled list updates:** For a scheduled call of the adblock service add an appropriate crontab entry (see example below)
  115. * **Change startup behaviour:** By default the startup will be triggered by the `wan` procd interface trigger. Choose `none` to disable automatic startups, `timed` to use a classic timeout (default 30 sec.) or select another trigger interface
  116. * **Suspend & resume adblocking:** To quickly switch the adblock service `on` or `off`, simply use `/etc/init.d/adblock [suspend|resume]`
  117. * **Domain query:** To query the active blocklist for a certain domain, please use the LuCI frontend or run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
  118. * **Add new list sources:** You can add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
  119. #### Further adblock config options
  120. * Usually the pre-configured adblock setup works quite well and no manual overrides are needed
  121. * The following options apply to the `global` config section:
  122. * `adb_enabled` => Main switch to enable/disable adblock service (default: `0`, disabled)
  123. * `adb_dns` => Select the DNS backend for your environment: `dnsmasq`, `unbound`, `named` or `kresd` (default: `dnsmasq`)
  124. * `adb_dnsvariant` => Select the blocking variant: `nxdomain` (default, supported by all backends), `null (IPv4)` and `null (IPv4/IPv6)` both options are only supported by `dnsmasq`
  125. * `adb_fetchutil` => Name of the used download utility: `uclient-fetch`, `wget`, `curl`, `aria2c`, `wget-nossl` or `busybox` (default: `uclient-fetch`)
  126. * `adb_fetchparm` => Special config options for the download utility (default: not set)
  127. * `adb_trigger` => Set the startup trigger to a certain interface, to `timed` or to `none` (default: `wan`)
  128. * The following options apply to the `extra` config section:
  129. * `adb_debug` => Enable/disable adblock debug output (default: `0`, disabled)
  130. * `adb_nice` => Set the nice level of the adblock process and all sub-processes (int/default: `0`, standard priority)
  131. * `adb_forcedns` => Force DNS requests to local resolver (bool/default: `0`, disabled)
  132. * `adb_maxqueue` => Size of the download queue to handle downloads & list processing in parallel (int/default: `8`)
  133. * `adb_dnsfilereset` => The final DNS blockfile will be purged after DNS backend loading to save storage space (bool/default: `false`, disabled)
  134. * `adb_report` => Enable the background tcpdump gathering process to provide a detailed DNS Query Report (bool/default: `0`, disabled)
  135. * `adb_repdir` => Target directory for DNS related report files generated by tcpdump (default: `/tmp`)
  136. * `adb_backupdir` => Target directory for adblock backups (default: `/tmp`)
  137. * `adb_mail` => Send notification emails in case of a processing errors or if the overall domain count is &le; 0 (bool/default: `0`, disabled)
  138. * `adb_mreceiver` => Receiver address for adblock notification emails (default: not set)
  139. * The following options could be added via "Additional Field" in LuCI and apply to the `extra` config section as well:
  140. * `adb_dnsdir` => Target directory for the generated blocklist `adb_list.overall` (default: not set, use DNS backend default)
  141. * `adb_dnsinstance` => set the relevant adblock enabled dnsmasq instance (int/default: '0', first dnsmasq instance)
  142. * `adb_blacklist` => Full path to the static blacklist file (default: `/etc/adblock/adblock.blacklist`)
  143. * `adb_whitelist` => Full path to the static whitelist file (default: `/etc/adblock/adblock.whitelist`)
  144. * `adb_triggerdelay` => Additional trigger delay in seconds before adblock processing begins (int/default: `2`)
  145. * `adb_maxtld` => Disable the tld compression, if the number of blocked domains is greater than this value (int/default: `100000`)
  146. * `adb_portlist` => Space separated list of fw ports which should be redirected locally (default: `53 853 5353`)
  147. * `adb_dnsinotify` => Disable adblock triggered restarts and the 'DNS File Reset' for DNS backends with autoload features (bool/default: `false`, disabled)
  148. * `adb_dnsflush` => Flush DNS cache after adblock processing, i.e. enable the old restart behavior (bool/default: `0`, disabled)
  149. * `adb_repiface` => Reporting interface used by tcpdump, set to `any` for multiple interfaces (default: `br-lan`)
  150. * `adb_replisten` => Space separated list of reporting port(s) used by tcpdump (default: `53`)
  151. * `adb_repchunkcnt` => Report chunk count used by tcpdump (default: `5`)
  152. * `adb_repchunksize` => Report chunk size used by tcpdump in MB (int/default: `1`)
  153. * `adb_msender` => Sender address for adblock notification emails (default: `no-reply@adblock`)
  154. * `adb_mtopic` => Topic for adblock notification emails (default: `adblock notification`)
  155. * `adb_mprofile` => Email profile used in `msmtp` for adblock notification emails (default: `adb_notify`)
  156. * `adb_mcnt` => Raise the minimum domain count email notification trigger (int/default: `0`)
  157. #### Examples
  158. **Change default DNS backend to `unbound`:**
  159. Adblock deposits the final blocklist `adb_list.overall` in `/var/lib/unbound` where unbound can find them in its jail, no further configuration needed.
  160. To preserve the DNS cache after adblock processing you need to install `unbound-control`.
  161. **Change default DNS backend to `named` (bind):**
  162. Adblock deposits the final blocklist `adb_list.overall` in `/var/lib/bind`.
  163. To preserve the DNS cache after adblock processing you need to install & configure `bind-rdnc`.
  164. To use the blocklist please modify `/etc/bind/named.conf`:
  165. * In the `options` namespace add:
  166. ```
  167. response-policy { zone "rpz"; };
  168. ```
  169. * And at the end of the file add:
  170. ```
  171. zone "rpz" {
  172. type master;
  173. file "/var/lib/bind/adb_list.overall";
  174. allow-query { none; };
  175. allow-transfer { none; };
  176. };
  177. ```
  178. **Change default DNS backend to `kresd`:**
  179. The knot-resolver (kresd) is only available on Turris Omnia devices.
  180. Adblock deposits the final blocklist `adb_list.overall` in `/etc/kresd`, no further configuration needed.
  181. **Enable email notification via msmtp:**
  182. To use the email notification you have to install & configure the package `msmtp`.
  183. Modify the file `/etc/msmtprc`:
  184. ```
  185. [...]
  186. defaults
  187. auth on
  188. tls on
  189. tls_certcheck off
  190. timeout 5
  191. syslog LOG_MAIL
  192. [...]
  193. account adb_notify
  194. host smtp.gmail.com
  195. port 587
  196. from dev.adblock@gmail.com
  197. user dev.adblock
  198. password xxx
  199. ```
  200. Finally enable email support and add a valid email address in LuCI.
  201. **Receive adblock runtime information:**
  202. ```
  203. /etc/init.d/adblock status
  204. ::: adblock runtime information
  205. + adblock_status : enabled
  206. + adblock_version : 3.8.0
  207. + overall_domains : 48359
  208. + fetch_utility : /bin/uclient-fetch (libustream-ssl)
  209. + dns_backend : dnsmasq, /tmp
  210. + dns_variant : null (IPv4/IPv6), true
  211. + backup_dir : /mnt/data/adblock
  212. + last_rundate : 15.08.2019 08:43:16
  213. + system_release : GL.iNet GL-AR750S, OpenWrt SNAPSHOT r10720-ccb4b96b8a
  214. ```
  215. **Receive adblock DNS Query Report information:**
  216. ```
  217. /etc/init.d/adblock report
  218. :::
  219. ::: Adblock DNS-Query Report
  220. :::
  221. + Start ::: 2018-12-19, 16:29:40
  222. + End ::: 2018-12-19, 16:45:08
  223. + Total ::: 42
  224. + Blocked ::: 17 (40.48 %)
  225. :::
  226. ::: Top 10 Clients
  227. + 32 ::: 101.167.1.103
  228. + 10 ::: abc1:abc1:abc0:0:abc1:abcb:abc5:abc3
  229. :::
  230. ::: Top 10 Domains
  231. + 7 ::: dns.msftncsi.com
  232. + 4 ::: forum.openwrt.org
  233. + 2 ::: outlook.office365.com
  234. + 1 ::: www.google.com
  235. + 1 ::: www.deepl.com
  236. + 1 ::: safebrowsing.googleapis.com
  237. + 1 ::: play.googleapis.com
  238. + 1 ::: odc.officeapps.live.com
  239. + 1 ::: login.microsoftonline.com
  240. + 1 ::: test-my.sharepoint.com
  241. :::
  242. ::: Top 10 Blocked Domains
  243. + 4 ::: nexus.officeapps.live.com
  244. + 4 ::: mobile.pipe.aria.microsoft.com
  245. + 3 ::: watson.telemetry.microsoft.com
  246. + 2 ::: v10.events.data.microsoft.com
  247. + 2 ::: settings-win.data.microsoft.com
  248. + 2 ::: nexusrules.officeapps.live.com
  249. [...]
  250. ```
  251. **Cronjob for regular block list updates (`/etc/crontabs/root`):**
  252. The following command as a cron job updates each individual block list from their source so that they hold the most current domains to block:
  253. ```
  254. 0 06 * * * /etc/init.d/adblock reload
  255. ```
  256. **Blacklist entry (`/etc/adblock/adblock.blacklist`):**
  257. ```
  258. ads.example.com
  259. This entry blocks the following (sub)domains:
  260. http://ads.example.com/foo.gif
  261. http://server1.ads.example.com/foo.gif
  262. https://ads.example.com:8000/
  263. This entry does not block:
  264. http://ads.example.com.ua/foo.gif
  265. http://example.com/
  266. ```
  267. **Whitelist entry (`/etc/adblock/adblock.whitelist`):**
  268. ```
  269. here.com
  270. This entry removes the following (sub)domains from the blocklist:
  271. maps.here.com
  272. here.com
  273. This entry does not remove:
  274. where.com
  275. www.adwhere.com
  276. ```
  277. **Query the active blocklist, the backups and black-/whitelist for a certain (sub-)domain, e.g. for whitelisting:**
  278. The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain. For every (sub-)domain it returns the first ten relevant results.
  279. ```
  280. /etc/init.d/adblock query google.com
  281. :::
  282. ::: results for domain 'google.com' in active blocklist
  283. :::
  284. + adservice.google.com
  285. + adservice.google.com.au
  286. + adservice.google.com.vn
  287. + adservices.google.com
  288. + analytics.google.com
  289. + googleadapis.l.google.com
  290. + pagead.l.google.com
  291. + partnerad.l.google.com
  292. + ssl-google-analytics.l.google.com
  293. + video-stats.video.google.com
  294. + [...]
  295. :::
  296. ::: results for domain 'google.com' in backups and black-/whitelist
  297. :::
  298. + adb_list.adguard.gz partnerad.l.google.com
  299. + adb_list.adguard.gz googleadapis.l.google.com
  300. + adb_list.adguard.gz ssl-google-analytics.l.google.com
  301. + adb_list.adguard.gz [...]
  302. + adb_list.disconnect.gz pagead.l.google.com
  303. + adb_list.disconnect.gz partnerad.l.google.com
  304. + adb_list.disconnect.gz video-stats.video.google.com
  305. + adb_list.disconnect.gz [...]
  306. + adb_list.whocares.gz video-stats.video.google.com
  307. + adb_list.whocares.gz adservice.google.com
  308. + adb_list.whocares.gz adservice.google.com.au
  309. + adb_list.whocares.gz [...]
  310. + adb_list.yoyo.gz adservice.google.com
  311. + adb_list.yoyo.gz analytics.google.com
  312. + adb_list.yoyo.gz pagead.l.google.com
  313. + adb_list.yoyo.gz [...]
  314. ```
  315. **Add a new blocklist source:**
  316. 1. The easy way ...
  317. Example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
  318. Adblock already supports an easylist source, called 'reg_ru'. To add the additional local easylist as a new source, copy the existing config source section and change only the source name, the url and the description - that's all!
  319. ```
  320. config source 'reg_ro'
  321. option enabled '0'
  322. option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
  323. option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([^([:space:]|#|\*|\/).]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
  324. option adb_src_desc 'focus on romanian ads plus generic easylist additions, weekly updates, approx. 9.400 entries'
  325. ```
  326. 2. A bit harder ...
  327. To add a really new source with different domain/host format you have to write a suitable awk one-liner on your own, so basic awk skills are needed. As a starting point check the already existing awk rulesets `adb_src_rset` in the config file, probably you need only small changes for your individual list. Download the desired list and test your new awk string locally. The output result should be a sequential list with one domain/host per line - nothing more. If your awk one-liner works quite well, add a new source section to the adblock config file and test the new source.
  328. ## Support
  329. Please join the adblock discussion in this [forum thread](https://forum.openwrt.org/t/adblock-support-thread/507) or contact me by email <dev@brenken.org>
  330. Have fun!
  331. Dirk