You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

247 lines
15 KiB

  1. # dns based ad/abuse domain blocking
  2. ## Description
  3. A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the dns server, a condition called the 'NXDOMAIN' occurred.
  4. ## Main Features
  5. * support of the following domain block list sources (free for private usage, for commercial use please check their individual licenses):
  6. * [adaway](https://adaway.org)
  7. * => infrequent updates, approx. 400 entries (enabled by default)
  8. * [blacklist]()
  9. * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  10. * [disconnect](https://disconnect.me)
  11. * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  12. * [dshield](http://dshield.org)
  13. * => daily updates, approx. 4.500 entries
  14. * [feodotracker](https://feodotracker.abuse.ch)
  15. * => daily updates, approx. 0-10 entries
  16. * [hphosts](https://hosts-file.net)
  17. * => monthly updates, approx. 50.000 entries
  18. * [malwaredomains](http://malwaredomains.com)
  19. * => daily updates, approx. 16.000 entries
  20. * [malwaredomainlist](http://www.malwaredomainlist.com)
  21. * => daily updates, approx. 1.500 entries
  22. * [openphish](https://openphish.com)
  23. * => numerous updates on the same day, approx. 1.800 entries
  24. * [palevo tracker](https://palevotracker.abuse.ch)
  25. * => daily updates, approx. 15 entries
  26. * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  27. * => daily updates, approx. 150 entries
  28. * [rolist/easylist](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  29. * => weekly updates, approx. 600 entries
  30. * [ruadlist/easylist](https://code.google.com/p/ruadlist)
  31. * => weekly updates, approx. 2.000 entries
  32. * [securemecca](http://www.securemecca.com)
  33. * => infrequent updates, approx. 25.000 entries
  34. * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  35. * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
  36. * [spam404](http://www.spam404.com)
  37. * => infrequent updates, approx. 5.000 entries
  38. * [sysctl/cameleon](http://sysctl.org/cameleon)
  39. * => weekly updates, approx. 21.000 entries
  40. * [whocares](http://someonewhocares.org)
  41. * => weekly updates, approx. 12.000 entries
  42. * [winhelp](http://winhelp2002.mvps.org)
  43. * => infrequent updates, approx. 15.000 entries
  44. * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  45. * => infrequent updates, approx. 120 entries
  46. * [yoyo](http://pgl.yoyo.org/adservers)
  47. * => weekly updates, approx. 2.500 entries (enabled by default)
  48. * [zeus tracker](https://zeustracker.abuse.ch)
  49. * => daily updates, approx. 440 entries
  50. * zero-conf like automatic installation & setup, usually no manual changes needed
  51. * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  52. * supports a wide range of router modes, even AP modes are supported
  53. * full IPv4 and IPv6 support
  54. * each block list source will be updated and processed separately
  55. * block list source parsing by fast & flexible regex rulesets
  56. * overall duplicate removal in separate block lists
  57. * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  58. * quality checks during block list update to ensure a reliable dns server service
  59. * minimal status & error logging to syslog, enable debug logging to receive more output
  60. * procd based init system support (start/stop/restart/reload/suspend/resume)
  61. * procd based hotplug support, the adblock start will be triggered by interface triggers
  62. * suspend & resume adblock actions temporarily without block list reloading
  63. * runtime statistics via ubus service call
  64. * query function to quickly identify blocked (sub-)domains, i.e. for whitelisting
  65. * automatic block list backup & restore, backups will be (de-)compressed and restored on the fly
  66. * add new adblock sources on your own via uci config
  67. ## Prerequisites
  68. * [openwrt](https://openwrt.org), tested with latest stable release (Chaos Calmer) and with current trunk (Designated Driver)
  69. * [LEDE project](https://www.lede-project.org), tested with trunk > r98
  70. * a usual setup with an enabled dns server at minimum - dump AP modes without a working dns server are _not_ supported
  71. * a download utility: 'wget', 'aria2c', 'uclient-fetch' or 'curl' are supported (only full versions with ssl support). Normally you should use the pre-configured default 'wget'. If you need a smaller memory footprint try 'uclient-fetch' without openssl dependency. The default ustream ssl backend 'libustream-polarssl' has issues with certain https sites and is currently not supported. To change the ssl backend see example below.
  72. ## OpenWrt / LEDE trunk Installation & Usage
  73. * install 'adblock' (_opkg install adblock_) and that's it - the adblock start will be automatically triggered by procd interface triggers
  74. * start/stop/restart/reload/suspend/resume the adblock service manually with _/etc/init.d/adblock_
  75. * enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
  76. ## LuCI adblock companion package
  77. * for easy management of the various block list sources and options you can also use a nice & efficient LuCI frontend
  78. * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
  79. * the application is located in LuCI under 'Services' menu
  80. * _Thanks to Hannu Nyman for this great adblock LuCI frontend!_
  81. ## Chaos Calmer installation notes
  82. * 'adblock' and 'luci-app-adblock' are _not_ available as .ipk packages in the Chaos Calmer download repository
  83. * download both packages from a development snapshot package directory:
  84. * for 'adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/packages/)
  85. * for 'luci-app-adblock' look [here](https://downloads.lede-project.org/snapshots/packages/x86_64/luci/)
  86. * manually transfer the packages to your routers temp directory (with tools like _sshfs_ or _winscp_)
  87. * install the packages with _opkg install <...>_ as described above
  88. ## Tweaks
  89. * **status/runtime statistics:** the adblock status and runtime statistics are available via ubus service call (see example below)
  90. * **debug output:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
  91. * **storage expansion:** to process and store all block list sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
  92. * **add white-/blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
  93. * **backup & restore block lists:** enable this feature, to restore automatically the latest compressed backup of your block lists in case of any processing error (i.e. a single block list source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
  94. * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
  95. * **restrict/disable procd interface trigger:** to restrict the procd interface trigger to a (list of) certain wan interface(s) or to disable it at all, set 'adb\_iface' to an existing interface like 'wan' or to a non-existing like 'false'
  96. * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
  97. * **domain query:** to query the active block lists for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
  98. * **divert dns requests:** to force dns requests to your local dns resolver add an appropriate firewall rule (see example below)
  99. * **add new list sources:** you could add new block list sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
  100. * **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
  101. ## Further adblock config options
  102. * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
  103. * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
  104. * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
  105. * adb\_iface => restrict the procd interface trigger to a (list of) certain wan interface(s) or disable it at all (default: not set, disabled)
  106. * adb\_fetch => reference an alternate download utility, see example below (default: not set, use wget)
  107. * adb\_fetchparm => set options for the download utility, see example below (default: not set, use wget options)
  108. ## Examples
  109. **example to change the ssl backend for 'uclient-fetch' or 'curl':**
  110. <pre><code>
  111. opkg update
  112. opkg remove --force-depends libustream-polarssl
  113. opkg install libustream-mbedtls
  114. </code></pre>
  115. **example configuration for different download utilities:**
  116. <pre><code>
  117. config for wget (default):
  118. option adb_fetch="/usr/bin/wget"
  119. option adb_fetchparm="--no-config --quiet --tries=1 --no-cache --no-cookies --max-redirect=0 --timeout=5 --no-check-certificate -O"
  120. config for aria2c:
  121. option adb_fetch '/usr/bin/aria2c'
  122. option adb_fetchparm '-q --max-tries=1 --timeout=5 --allow-overwrite=true --auto-file-renaming=false --check-certificate=false -o'
  123. config for uclient-fetch (download errors with default ssl backend!):
  124. option adb_fetch '/bin/uclient-fetch'
  125. option adb_fetchparm '-q --timeout=5 --no-check-certificate -O'
  126. config for curl (download errors with default ssl backend!):
  127. option adb_fetch '/usr/bin/curl'
  128. option adb_fetchparm '-s --retry 1 --connect-timeout 5 --insecure -o'
  129. </code></pre>
  130. **example to receive adblock statistics via ubus:**
  131. <pre><code>
  132. ubus call service list '{"name":"adblock_stats"}' | jsonfilter -e '@.*.instances.stats.env'
  133. This will output the overall domain count and the last runtime as JSON, i.e. { "blocked_domains": "136159", "last_rundate": "18.12.2016 20:49:03" }
  134. </code></pre>
  135. **example cronjob for a regular block list update (/etc/crontabs/root):**
  136. <pre><code>
  137. 0 06 * * * /etc/init.d/adblock start
  138. </code></pre>
  139. **example blacklist entry (/etc/adblock/adblock.blacklist):**
  140. <pre><code>
  141. ads.example.com
  142. This entry blocks the following (sub)domains:
  143. http://ads.example.com/foo.gif
  144. http://server1.ads.example.com/foo.gif
  145. https://ads.example.com:8000/
  146. This entry does not block:
  147. http://ads.example.com.ua/foo.gif
  148. http://example.com/
  149. </code></pre>
  150. **example whitelist entry (/etc/adblock/adblock.whitelist):**
  151. <pre><code>
  152. here.com
  153. This entry removes the following (sub)domains from the block lists:
  154. maps.here.com
  155. here.com
  156. This entry does not remove:
  157. where.com
  158. www.adwhere.com
  159. </code></pre>
  160. **example to query active block lists for a certain (sub-)domain, i.e. for whitelisting:**
  161. <pre><code>
  162. /etc/init.d/adblock query "example.www.doubleclick.net"
  163. :: distinct results for domain 'example.www.doubleclick.net' (overall 0)
  164. no matches in active block lists
  165. :: distinct results for domain 'www.doubleclick.net' (overall 1)
  166. adb_list.securemecca : www.doubleclick.net
  167. :: distinct results for domain 'doubleclick.net' (overall 127)
  168. adb_list.adaway : ad-g.doubleclick.net
  169. adb_list.securemecca : 1168945.fls.doubleclick.net
  170. The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain(s).
  171. For every domain it returns the overall count plus a distinct list of active block lists with the first relevant result.
  172. In the example above whitelist "www.doubleclick.net" to free the submitted domain.
  173. </code></pre>
  174. **example to divert dns requests to local dns resolver (/etc/config/firewall):**
  175. <pre><code>
  176. config redirect
  177. option name 'Divert DNS'
  178. option src 'lan'
  179. option proto 'tcp udp'
  180. option src_dport '53'
  181. option dest_port '53'
  182. option target 'DNAT'
  183. </code></pre>
  184. **example to add a new block list source:**
  185. <pre><code>
  186. 1. the easy way ...
  187. example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
  188. adblock already supports an easylist source, called 'ruadlist'. To add the additional local easylist
  189. as a new source, copy the existing config source 'ruadlist' section and change only
  190. the source name, the url and the description - that's all!
  191. config source 'rolist'
  192. option enabled '0'
  193. option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
  194. option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
  195. option adb_src_desc 'focus on romanian ad related domains plus generic easylist additions, weekly updates, approx. 600 entries'
  196. 2. a bit harder ...
  197. to add a really new source with different domain/host format you have to write a suitable
  198. awk one-liner on your own, so basic awk skills are needed. As a starting point check the already
  199. existing awk strings (adb_src_rset) in adblock config, maybe you need only small changes for your individual list.
  200. Download the desired list and test your new awk string locally with:
  201. cat new.list | awk 'fs__individual search__search core__result'
  202. 'fs' => field separator (optional)
  203. 'individual search' => individual search part to filter out needless list information
  204. 'search core' => always '([A-Za-z0-9_-]+\.){1,}[A-Za-z]+', this is part of all list sources and should be unchanged
  205. 'result' => always '{print tolower(\$n)}', only the output column 'n' may vary
  206. the output result should be a sequential list with one domain/host per line - nothing more.
  207. If your awk one-liner works quite well, add a new source section in adblock config and test your new source
  208. </code></pre>
  209. ## Support
  210. Please join the adblock discussion in this [forum thread](https://forum.openwrt.org/viewtopic.php?id=59803) or contact me by mail <dev@brenken.org>
  211. ## Removal
  212. * stop all adblock related services with _/etc/init.d/adblock stop_
  213. * optional: remove the adblock package (_opkg remove adblock_)
  214. Have fun!
  215. Dirk