You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

312 lines
19 KiB

  1. # dns based ad/abuse domain blocking
  2. ## Description
  3. A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the dns server, a condition called the 'NXDOMAIN' occurred.
  4. ## Main Features
  5. * support of the following domain blocklist sources (free for private usage, for commercial use please check their individual licenses):
  6. * [adaway](https://adaway.org)
  7. * => infrequent updates, approx. 400 entries (enabled by default)
  8. * [adguard](https://adguard.com)
  9. * => numerous updates on the same day, approx. 12.000 entries
  10. * [bitcoin](https://github.com/hoshsadiq/adblock-nocoin-list)
  11. * => infrequent updates, approx. 15 entries
  12. * [blacklist]()
  13. * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  14. * [disconnect](https://disconnect.me)
  15. * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  16. * [dshield](http://dshield.org)
  17. * => daily updates, approx. 4.500 entries
  18. * [feodotracker](https://feodotracker.abuse.ch)
  19. * => daily updates, approx. 0-10 entries
  20. * [hphosts](https://hosts-file.net)
  21. * => monthly updates, approx. 50.000 entries
  22. * [malwaredomains](http://malwaredomains.com)
  23. * => daily updates, approx. 16.000 entries
  24. * [malwaredomainlist](http://www.malwaredomainlist.com)
  25. * => daily updates, approx. 1.500 entries
  26. * [openphish](https://openphish.com)
  27. * => numerous updates on the same day, approx. 1.800 entries
  28. * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  29. * => daily updates, approx. 150 entries
  30. * [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt)
  31. * => regional blocklist for China, daily updates, approx. 1.600 entries
  32. * [reg_cz](https://raw.githubusercontent.com/qxstyles/turris-hole-czech-block-list/master/turris-hole-czech-block-list)
  33. * => regional blocklist for Czechia, maintained by Turris Omnia Users, infrequent updates, approx. 100 entries
  34. * [reg_de](https://easylist-downloads.adblockplus.org/easylistgermany+easylist.txt)
  35. * => regional blocklist for Germany, daily updates, approx. 9.200 entries
  36. * [reg_id](https://easylist-downloads.adblockplus.org/abpindo+easylist.txt)
  37. * => regional blocklist for Indonesia, daily updates, approx. 800 entries
  38. * [reg_nl](https://easylist-downloads.adblockplus.org/easylistdutch+easylist.txt)
  39. * => regional blocklist for the Netherlands, weekly updates, approx. 1300 entries
  40. * [reg_pl](http://adblocklist.org)
  41. * => regional blocklist for Poland, daily updates, approx. 50 entries
  42. * [reg_ro](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  43. * => regional blocklist for Romania, weekly updates, approx. 600 entries
  44. * [reg_ru](https://code.google.com/p/ruadlist)
  45. * => regional blocklist for Russia, weekly updates, approx. 2.000 entries
  46. * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  47. * => daily updates, approx. 32.000 entries (a short description of all categories can be found [online](http://www.shallalist.de/categories.html))
  48. * [spam404](http://www.spam404.com)
  49. * => infrequent updates, approx. 5.000 entries
  50. * [sysctl/cameleon](http://sysctl.org/cameleon)
  51. * => weekly updates, approx. 21.000 entries
  52. * [ut_capitole](https://dsi.ut-capitole.fr/blacklists) (categories "cryptojacking" "ddos" "malware" "phishing" "warez" enabled by default)
  53. * => daily updates, approx. 64.000 entries (a short description of all categories can be found [online](https://dsi.ut-capitole.fr/blacklists/index_en.php))
  54. * [urlhaus](https://urlhaus.abuse.ch)
  55. * => numerous updates on the same day, approx. 3.500 entries
  56. * [whocares](http://someonewhocares.org)
  57. * => weekly updates, approx. 12.000 entries
  58. * [winhelp](http://winhelp2002.mvps.org)
  59. * => infrequent updates, approx. 15.000 entries
  60. * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  61. * => infrequent updates, approx. 120 entries
  62. * [youtube](https://api.hackertarget.com/hostsearch/?q=googlevideo.com)
  63. * => dynamic request API to filter "random" youtube ad domains (experimental!), approx. 150 entries
  64. * [yoyo](http://pgl.yoyo.org/adservers)
  65. * => weekly updates, approx. 2.500 entries (enabled by default)
  66. * [zeus tracker](https://zeustracker.abuse.ch)
  67. * => daily updates, approx. 440 entries
  68. * zero-conf like automatic installation & setup, usually no manual changes needed
  69. * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  70. * supports five different dns backends / blocklist formats: dnsmasq, unbound, named (bind), kresd and dnscrypt-proxy
  71. * supports six different download utilities: uclient-fetch, wget, curl, aria2c, wget-nossl, busybox-wget
  72. * Really fast downloads & list processing as they are handled in parallel as background jobs in a configurable 'Download Queue'
  73. * provides 'http only' mode without installed ssl library for all non-SSL blocklist sources
  74. * supports a wide range of router modes, even AP modes are supported
  75. * full IPv4 and IPv6 support
  76. * provides top level domain compression ('tld compression'), this feature removes thousands of needless host entries from the blocklist and lowers the memory footprint for the dns backend
  77. * blocklist source parsing by fast & flexible regex rulesets
  78. * overall duplicate removal in central blocklist 'adb_list.overall'
  79. * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  80. * quality checks during blocklist update to ensure a reliable dns backend service
  81. * minimal status & error logging to syslog, enable debug logging to receive more output
  82. * procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
  83. * procd network interface trigger support or classic time based startup
  84. * keep the dns cache intact after adblock processing (currently supported by unbound, named and kresd)
  85. * conditional dns backend restarts by old/new blocklist comparison with sha256sum (default) or md5sum
  86. * suspend & resume adblock actions temporarily without blocklist reloading
  87. * output comprehensive runtime information via LuCI or via 'status' init command
  88. * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
  89. * strong LuCI support
  90. * optional: force dns requests to local resolver
  91. * optional: force overall sort / duplicate removal for low memory devices (handle with care!)
  92. * optional: automatic blocklist backup & restore, they will be used in case of download errors or during startup in backup mode
  93. * optional: 'backup mode' to re-use blocklist backups during startup, get fresh lists only via reload or restart action
  94. * optional: 'Jail' blocklist generation which builds an additional list (/tmp/adb_list.jail) to block access to all domains except those listed in the whitelist file. You can use this restrictive blocklist manually e.g. for guest wifi or kidsafe configurations
  95. * optional: send notification emails in case of a processing error or if the overall domain count is ≤ 0
  96. * optional: add new adblock sources on your own, see example below
  97. ## Prerequisites
  98. * [OpenWrt](https://openwrt.org), tested with the stable release series (18.06) and with the latest snapshot
  99. * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
  100. * a download utility:
  101. * to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
  102. * for limited devices with real memory constraints, adblock provides also a 'http only' option and supports wget-nossl and uclient-fetch (without libustream-ssl) as well
  103. * for more configuration options see examples below
  104. ## Installation & Usage
  105. * install 'adblock' (_opkg install adblock_)
  106. * at minimum configure the appropriate dns backend ('dnsmasq' by default), the download utility and enable the adblock service in _/etc/config/adblock_
  107. * control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
  108. ## LuCI adblock companion package
  109. * for easy management of the various blocklist sources and adblock runtime options you should use the provided LuCI frontend
  110. * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
  111. * the application is located in LuCI under 'Services' menu
  112. ## Tweaks
  113. * **runtime information:** the adblock status is available via _/etc/init.d/adblock status_ (see example below)
  114. * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
  115. * **storage expansion:** to process and store all blocklist sources at once it might helpful to enlarge your temp directory with a swap partition => see [OpenWrt Wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
  116. * **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
  117. * **backup & restore blocklists:** enable this feature, to restore automatically the latest compressed backup of your blocklists in case of any processing error (e.g. a single blocklist source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
  118. * **download queue size:** for further download & list processing performance improvements you can raise the 'adb\_maxqueue' value, e.g. '8' or '16' should be safe
  119. * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
  120. * **change startup behaviour:** by default the startup will be triggered by the 'wan' procd interface trigger. Choose 'none' to disable automatic startups, 'timed' to use a classic timeout (default 30 sec.) or select another trigger interface
  121. * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
  122. * **domain query:** to query the active blocklist for a certain domain, please use the LuCI frontend or run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
  123. * **add new list sources:** you could add new blocklist sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
  124. * **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
  125. ## Further adblock config options
  126. * usually the pre-configured adblock setup works quite well and no manual overrides are needed
  127. * the following options apply to the 'global' config section:
  128. * adb\_enabled => main switch to enable/disable adblock service (default: '0', disabled)
  129. * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
  130. * adb\_fetchutil => name of the used download utility: 'uclient-fetch', 'wget', 'curl', 'aria2c', 'wget-nossl'. 'busybox' (default: 'uclient-fetch')
  131. * adb\_fetchparm => special config options for the download utility (default: not set)
  132. * adb\_dns => select the dns backend for your environment: 'dnsmasq', 'unbound', 'named', 'kresd' or 'dnscrypt-proxy' (default: 'dnsmasq')
  133. * adb\_dnsdir => target directory for the generated blocklist 'adb_list.overall' (default: not set, use dns backend default)
  134. * adb\_trigger => set the startup trigger to a certain interface, to 'timed' or to 'none' (default: 'wan')
  135. * the following options apply to the 'extra' config section:
  136. * adb\_triggerdelay => additional trigger delay in seconds before adblock processing begins (int/default: '2')
  137. * adb\_forcedns => force dns requests to local resolver (bool/default: '0', disabled)
  138. * adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (bool/default: '0', disabled)
  139. * adb\_backup_mode => do not automatically update blocklists during startup, use backups instead (bool/default: '0', disabled)
  140. * adb\_maxqueue => size of the download queue to handle downloads & list processing in parallel (int/default: '4')
  141. * adb\_jail => builds an additional 'Jail' list (/tmp/adb_list.jail) to block access to all domains except those listed in the whitelist file (bool/default: '0', disabled)
  142. * adb\_dnsflush => flush DNS cache after adblock processing, i.e. enable the old restart behavior (bool/default: '0', disabled)
  143. * adb\_notify => send notification emails in case of a processing error or if the overall domain count is &le; 0 (bool/default: '0', disabled)
  144. * adb\_notifycnt => Raise minimum domain count email notification trigger (int/default: '0')
  145. ## Examples
  146. **change default dns backend to 'unbound':**
  147. Adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/unbound' where unbound can find them in its jail.
  148. To preserve the DNS cache after adblock processing you need to install 'unbound-control'.
  149. **change default dns backend to 'named' (bind):**
  150. Adblock deposits the final blocklist 'adb_list.overall' in '/var/lib/bind'.
  151. To preserve the DNS cache after adblock processing you need to install & configure 'bind-rdnc'.
  152. To use the blocklist please modify '/etc/bind/named.conf':
  153. <pre><code>
  154. in the 'options' namespace add:
  155. response-policy { zone "rpz"; };
  156. and at the end of the file add:
  157. zone "rpz" {
  158. type master;
  159. file "/var/lib/bind/adb_list.overall";
  160. allow-query { none; };
  161. allow-transfer { none; };
  162. };
  163. </code></pre>
  164. **change default dns backend to 'kresd':**
  165. The knot-resolver (kresd) is only available on Turris Omnia devices.
  166. Adblock deposits the final blocklist 'adb_list.overall' in '/etc/kresd', no further configuration needed.
  167. **change default dns backend to 'dnscrypt-proxy':**
  168. The required 'blacklist' option of dnscrypt-proxy is not enabled by default, because the package will be compiled without plugins support.
  169. Take a custom OpenWrt build with plugins support to use this feature. Adblock deposits the final blocklist 'adb_list.overall' in '/tmp'.
  170. To use the blocklist please modify '/etc/config/dnscrypt-proxy' per instance:
  171. <pre><code>
  172. list blacklist 'domains:/tmp/adb_list.overall'
  173. </code></pre>
  174. **enable email notification via msmtp:**
  175. To use the email notification you have to install & configure the package 'msmtp'.
  176. Modify the file '/etc/msmtprc':
  177. <pre><code>
  178. [...]
  179. defaults
  180. auth on
  181. tls on
  182. tls_certcheck off
  183. timeout 5
  184. syslog LOG_MAIL
  185. [...]
  186. account adb_notify
  187. host smtp.gmail.com
  188. port 587
  189. from dev.adblock@gmail.com
  190. user dev.adblock
  191. password xxx
  192. </code></pre>
  193. Edit the file '/etc/adblock/adblock.notify' and change at least the 'mail_receiver'.
  194. Finally make this file executable via 'chmod' and test it directly. If no more errors come up you can comment 'mail_debug', too.
  195. **receive adblock runtime information:**
  196. <pre><code>
  197. /etc/init.d/adblock status
  198. ::: adblock runtime information
  199. + adblock_status : enabled
  200. + adblock_version : 3.5.5
  201. + overall_domains : 97199 (backup mode)
  202. + fetch_utility : /bin/uclient-fetch (libustream-ssl)
  203. + dns_backend : unbound (/var/lib/unbound)
  204. + last_rundate : 01.09.2018 07:09:16
  205. + system_release : PC Engines APU, OpenWrt SNAPSHOT r7986-dc9388ac55
  206. </code></pre>
  207. **cronjob for a regular block list update (/etc/crontabs/root):**
  208. <pre><code>
  209. 0 06 * * * /etc/init.d/adblock reload
  210. </code></pre>
  211. **blacklist entry (/etc/adblock/adblock.blacklist):**
  212. <pre><code>
  213. ads.example.com
  214. This entry blocks the following (sub)domains:
  215. http://ads.example.com/foo.gif
  216. http://server1.ads.example.com/foo.gif
  217. https://ads.example.com:8000/
  218. This entry does not block:
  219. http://ads.example.com.ua/foo.gif
  220. http://example.com/
  221. </code></pre>
  222. **whitelist entry (/etc/adblock/adblock.whitelist):**
  223. <pre><code>
  224. here.com
  225. This entry removes the following (sub)domains from the blocklist:
  226. maps.here.com
  227. here.com
  228. This entry does not remove:
  229. where.com
  230. www.adwhere.com
  231. </code></pre>
  232. **query the active blocklist for a certain (sub-)domain, e.g. for whitelisting:**
  233. The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain. For every (sub-)domain it returns the first ten relevant results.
  234. <pre><code>
  235. /etc/init.d/adblock query www.example.google.com
  236. ::: results for domain 'www.example.google.com'
  237. - no match
  238. ::: results for domain 'example.google.com'
  239. - no match
  240. ::: results for domain 'google.com'
  241. + ads.google.com
  242. + adservices.google.com
  243. + adwords.google.com
  244. + ampcid.google.com
  245. + analytics.google.com
  246. + gg.google.com
  247. + google.com.analytics.kdgsrltkcun.com
  248. + googleadapis.l.google.com
  249. + id.google.com
  250. + pagead-googlehosted.l.google.com
  251. + [...]
  252. </code></pre>
  253. **add a new blocklist source:**
  254. 1. the easy way ...
  255. example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
  256. Adblock already supports an easylist source, called 'reg_ru'. To add the additional local easylist as a new source, copy the existing config source section and change only
  257. the source name, the url and the description - that's all!
  258. <pre><code>
  259. config source 'reg_ro'
  260. option enabled '0'
  261. option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
  262. option adb_src_rset 'BEGIN{FS=\"[|^]\"}/^\|\|([^([:space:]|#|\*|\/).]+\.)+[[:alpha:]]+\^("\\\$third-party")?$/{print tolower(\$3)}'
  263. option adb_src_desc 'focus on romanian ads plus generic easylist additions, weekly updates, approx. 9.400 entries'
  264. </code></pre>
  265. 2. a bit harder ...
  266. To add a really new source with different domain/host format you have to write a suitable awk one-liner on your own, so basic awk skills are needed. As a starting point check the already existing awk rulesets 'adb_src_rset' in the config file, probably you need only small changes for your individual list. Download the desired list and test your new awk string locally. The output result should be a sequential list with one domain/host per line - nothing more. If your awk one-liner works quite well, add a new source section to the adblock config file and test the new source.
  267. ## Support
  268. Please join the adblock discussion in this [forum thread](https://forum.lede-project.org/t/adblock-2-x-support-thread/507) or contact me by mail <dev@brenken.org>
  269. ## Removal
  270. * stop all adblock related services with _/etc/init.d/adblock stop_
  271. * optional: remove the adblock package (_opkg remove adblock_)
  272. Have fun!
  273. Dirk