Browse Source

adblock: maintenance update

* remove "downloads" category from shallalist default configuration, due
to false positives (i.e. debian & microsoft update sites)
* remove example white- & blacklist entries
* further optimize shallalist handling
* fix domain merging with empty whitelist
* various minor fixes & enhancements
* documentation update


Signed-off-by: Dirk Brenken <dirk@brenken.org>
lilik-openwrt-22.03
Dirk Brenken 9 years ago
parent
commit
45f005d856
7 changed files with 63 additions and 64 deletions
  1. +2
    -2
      net/adblock/Makefile
  2. +27
    -25
      net/adblock/files/README.md
  3. +17
    -16
      net/adblock/files/adblock-helper.sh
  4. +15
    -15
      net/adblock/files/adblock-update.sh
  5. +1
    -1
      net/adblock/files/adblock.blacklist
  6. +0
    -1
      net/adblock/files/adblock.conf
  7. +1
    -4
      net/adblock/files/adblock.whitelist

+ 2
- 2
net/adblock/Makefile View File

@ -7,7 +7,7 @@
include $(TOPDIR)/rules.mk
PKG_NAME:=adblock
PKG_VERSION:=0.22.1
PKG_VERSION:=0.22.2
PKG_RELEASE:=1
PKG_LICENSE:=GPL-3.0+
PKG_MAINTAINER:=Dirk Brenken <dirk@brenken.org>
@ -17,7 +17,7 @@ include $(INCLUDE_DIR)/package.mk
define Package/$(PKG_NAME)
SECTION:=net
CATEGORY:=Network
TITLE:=powerful adblock script to block ad/abuse domains
TITLE:=Powerful adblock script to block ad/abuse domains
DEPENDS:=+curl +wget
PKGARCH:=all
endef


+ 27
- 25
net/adblock/files/README.md View File

@ -12,22 +12,24 @@ and return the local ip address of your router and the internal web server deliv
* support of the following domain blacklist sites (free for private usage, for commercial use please check their individual licenses):
* [pgl.yoyo.org](http://pgl.yoyo.org/adservers), approx. 2.500 entries
* [malwaredomains.com](http://malwaredomains.com), approx. 16.000 entries
* [zeustracker.abuse.ch](https://zeustracker.abuse.ch), currently down
* [zeustracker.abuse.ch](https://zeustracker.abuse.ch), approx. 420 entries
* [feodotracker.abuse.ch](https://feodotracker.abuse.ch), approx. 10 entries
* [palevotracker.abuse.ch](https://palevotracker.abuse.ch), approx. 10 entries
* [dshield.org](http://dshield.org), approx. 4.500 entries
* [shallalist.de](http://www.shallalist.de) (tested with the categories "adv" "costtraps" "downloads" "spyware" "tracker" "warez"), approx. 37.000 entries
* [shallalist.de](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default), approx. 32.000 entries
* a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html)
* [spam404.com](http://www.spam404.com), approx. 5.000 entries
* [winhelp2002.mvps.org](http://winhelp2002.mvps.org), approx. 15.000 entries
* blocklist parsing by fast & flexible regex rulesets
* additional white- and blacklist support for manual overrides
* separate dynamic adblock network interface
* separate dynamic uhttpd instance as pixel server
* optional: quality checks and a powerful backup/restore handling to ensure a reliable dnsmasq service
* adblock quality checks after list update to ensure a reliable dnsmasq service
* optional: powerful adblock list backup/restore handling
* optional: adblock updates only on pre-defined wan interfaces (useful for (mobile) multiwan setups)
* optional: domain query logging as a background service to easily identify free and already blocked domains
* optional: domain query logging as a background service to easily identify free and already blocked domains (see example output below)
* optional: status & error logging to separate file (req. ntp time sync)
* optional: ntp time sync
* optional: status & error logging (req. ntp time sync)
## Prerequisites
* [openwrt](https://openwrt.org) (tested only with trunk > r47025), CC should also work
@ -41,8 +43,9 @@ and return the local ip address of your router and the internal web server deliv
## Usage
* select & install adblock package (*opkg install adblock*)
* configure /etc/config/adblock to your needs, see additional comments in *adblock.conf.sample*
* configure */etc/config/adblock* to your needs, see additional comments in *adblock.conf.sample*
* at least configure the ip address of the local adblock interface/uhttpd instance, needs to be a different subnet from the normal LAN
* optional: add additional domain white- or blacklist entries, one domain per line (wildcards & regex are not allowed!), both list are located in */etc/adblock*
* by default openwrts main uhttpd instance is bind to all ports of your router. For a working adblock setup you have to bind uhttpd to the standard LAN port only, please change listen_http accordingly
* start /usr/bin/adblock-update.sh and check console output or *logread -e "adblock"* for errors
@ -59,25 +62,24 @@ and return the local ip address of your router and the internal web server deliv
stdout excerpt for successful adblock run:
root@pi2wrt:~# /usr/bin/adblock-update.sh
adblock[17771] info : domain adblock processing started (0.21.0)
adblock[17771] info : get wan/update interface (wlan1), after 0 loops
adblock[17771] info : get ntp time sync (0.pool.ntp.org 1.pool.ntp.org 2.pool.ntp.org 3.pool.ntp.org), after 0 loops
adblock[17771] info : shallalist archive download finished
adblock[17771] info : shallalist archive extraction finished
adblock[17771] info : shallalist (pre-)processing finished (adv costtraps downloads spyware tracker warez)
adblock[17771] info : source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2426 entries)
adblock[17771] info : source download finished (http://mirror1.malwaredomains.com/files/justdomains, 15275 entries)
adblock[17771] info : source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 3 entries)
adblock[17771] info : source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)
adblock[17771] info : source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 11 entries)
adblock[17771] info : source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)
adblock[17771] info : source download finished (http://spam404bl.com/spam404scamlist.txt, 5193 entries)
adblock[17771] info : source download finished (http://winhelp2002.mvps.org/hosts.txt, 13852 entries)
adblock[17771] info : source download finished (file:////tmp/tmp.emlDeH/shallalist.txt, 36961 entries)
adblock[17771] info : source download finished (file:///etc/adblock/adblock.blacklist, 1 entries)
adblock[17771] info : new adblock list with 73090 domains loaded, backup generated
adblock[17771] info : domain adblock processing finished (0.21.0)
adblock[11541] info : domain adblock processing started (0.22.2, r47665, 29.11.2015 14:58:11)
adblock[11541] info : wan update check will be disabled
adblock[11541] info : get ntp time sync (192.168.254.254), after 0 loops
adblock[11541] info : shallalist (pre-)processing started ...
adblock[11541] info : shallalist (pre-)processing finished (adv costtraps spyware tracker warez)
adblock[11541] info : source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2423 entries)
adblock[11541] info : source download finished (http://mirror1.malwaredomains.com/files/justdomains, 16016 entries)
adblock[11541] info : source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 419 entries)
adblock[11541] info : source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)
adblock[11541] info : source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 12 entries)
adblock[11541] info : source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)
adblock[11541] info : source download finished (http://spam404bl.com/spam404scamlist.txt, 5193 entries)
adblock[11541] info : source download finished (http://winhelp2002.mvps.org/hosts.txt, 13635 entries)
adblock[11541] info : source download finished (file:////tmp/tmp.CgbMmO/shallalist.txt, 32446 entries)
adblock[11541] info : empty source download finished (file:///etc/adblock/adblock.blacklist)
adblock[11541] info : domain merging finished
adblock[11541] info : new adblock list with 69646 domains loaded, backup generated
adblock[11541] info : domain adblock processing finished (0.22.2, r47665, 29.11.2015 14:59:23)
generated domain blocklist for dnsmasq:


+ 17
- 16
net/adblock/files/adblock-helper.sh View File

@ -195,7 +195,7 @@ f_envcheck()
# check main uhttpd configuration
#
check_uhttpd="$(uci get uhttpd.main.listen_http 2>/dev/null | grep -o "0.0.0.0")"
check_uhttpd="$(uci get uhttpd.main.listen_http 2>/dev/null | grep -Fo "0.0.0.0")"
if [ -n "${check_uhttpd}" ]
then
rc=530
@ -287,20 +287,20 @@ f_envcheck()
if [ -z "${check}" ]
then
curl_parm="-q --insecure"
wget_parm="--no-config --no-check-certificate"
wget_parm="--no-config --no-hsts --no-check-certificate"
else
curl_parm="-q"
wget_parm="--no-config"
wget_parm="--no-config --no-hsts"
fi
# check total and swap memory
#
mem_total="$(cat /proc/meminfo | grep "MemTotal" | grep -o "[0-9]*")"
mem_free="$(cat /proc/meminfo | grep "MemFree" | grep -o "[0-9]*")"
swap_total="$(cat /proc/meminfo | grep "SwapTotal" | grep -o "[0-9]*")"
mem_total="$(cat /proc/meminfo | grep -F "MemTotal" | grep -o "[0-9]*")"
mem_free="$(cat /proc/meminfo | grep -F "MemFree" | grep -o "[0-9]*")"
swap_total="$(cat /proc/meminfo | grep -F "SwapTotal" | grep -o "[0-9]*")"
if [ $((mem_total)) -le 64000 ] && [ $((swap_total)) -eq 0 ]
then
f_log "please consider to add an external swap device to supersize your /tmp directory (total: ${mem_total}, free: ${mem_free}, swap: ${mem_swap})"
f_log "please consider adding an external swap device to supersize your /tmp directory (total: ${mem_total}, free: ${mem_free}, swap: ${mem_swap})"
fi
# check backup configuration
@ -323,7 +323,7 @@ f_envcheck()
then
# check find capabilities
#
check="$(find --help 2>&1 | grep "mtime")"
check="$(find --help 2>&1 | grep -F "mtime")"
if [ -z "${check}" ]
then
query_ok="false"
@ -340,7 +340,7 @@ f_envcheck()
if [ -s "${adb_querypid}" ]
then
kill -9 "$(cat "${adb_querypid}")" >/dev/null 2>&1
f_log "remove old dns query log background process (pid: $(cat "${adb_querypid}"))"
f_log "remove old dns query log background process (pid: $(cat "${adb_querypid}" 2>/dev/null))"
> "${adb_querypid}"
fi
fi
@ -442,11 +442,12 @@ f_log()
then
class="error"
log_rc=", rc: ${log_rc}"
log_msg="${log_msg}${log_rc}"
fi
/usr/bin/logger -s -t "adblock[${pid}] ${class}" "${log_msg}${log_rc}"
/usr/bin/logger -s -t "adblock[${pid}] ${class}" "${log_msg}"
if [ "${log_ok}" = "true" ] && [ "${ntp_ok}" = "true" ]
then
printf "%s\n" "$(/bin/date "+%d.%m.%Y %H:%M:%S") adblock[${pid}] ${class}: ${log_msg}${log_rc}" >> "${adb_logfile}"
printf "%s\n" "$(/bin/date "+%d.%m.%Y %H:%M:%S") adblock[${pid}] ${class}: ${log_msg}" >> "${adb_logfile}"
fi
fi
}
@ -500,7 +501,7 @@ f_deltemp()
then
rm -rf "${adb_tmpdir}" >/dev/null 2>&1
fi
f_log "domain adblock processing finished (${adb_version}, ${openwrt_version})"
f_log "domain adblock processing finished (${adb_version}, ${openwrt_version}, $(/bin/date "+%d.%m.%Y %H:%M:%S"))"
exit ${rc}
}
@ -516,13 +517,13 @@ f_remove()
then
kill -9 "$(cat "${adb_querypid}")" >/dev/null 2>&1
find "${adb_backupdir}" -maxdepth 1 -type f -mtime +"${adb_queryhistory}" -name "${query_name}.*" -exec rm -f {} \; 2>/dev/null
f_log "remove old dns query log background process (pid: $(cat "${adb_querypid}")) and do logfile housekeeping"
f_log "remove old domain query log background process (pid: $(cat "${adb_querypid}")) and do logfile housekeeping"
> "${adb_querypid}"
fi
if [ ! -s "${adb_querypid}" ]
then
( logread -f 2>/dev/null & printf ${!} > "${adb_querypid}" ) | egrep -o "(query\[A\].*)|([a-z0-9\.\-]* is ${query_ip}$)" >> "${adb_queryfile}.${query_date}" &
f_log "new domain query log background process started (pid: $(cat "${adb_querypid}"))"
(logread -f 2>/dev/null & printf ${!} > "${adb_querypid}") | grep -Eo "(query\[A\].*)|([a-z0-9\.\-]* is ${query_ip}$)" >> "${adb_queryfile}.${query_date}" &
f_log "new domain query log background process started"
fi
fi
f_deltemp
@ -626,7 +627,7 @@ f_dnscheck()
rc=${?}
if [ -z "${dns_status}" ]
then
dns_status="$(nslookup "${adb_domain}" 2>/dev/null | grep "${adb_ip}")"
dns_status="$(nslookup "${adb_domain}" 2>/dev/null | grep -F "${adb_ip}")"
rc=${?}
if [ -z "${dns_status}" ]
then


+ 15
- 15
net/adblock/files/adblock-update.sh View File

@ -25,7 +25,7 @@
# set script version
#
adb_version="0.22.1"
adb_version="0.22.2"
# get current pid, script directory and openwrt version
#
@ -54,7 +54,7 @@ trap "f_log 'trap error' '600'; f_restore" 1 2 3 10 11 15
# start logging
#
f_log "domain adblock processing started (${adb_version}, ${openwrt_version})"
f_log "domain adblock processing started (${adb_version}, ${openwrt_version}, $(/bin/date "+%d.%m.%Y %H:%M:%S"))"
# load environment
#
@ -74,14 +74,13 @@ if [ -n "${adb_arc_shalla}" ]
then
# download shallalist archive
#
f_log "shallalist (pre-)processing started ..."
shalla_archive="${adb_tmpdir}/shallalist.tar.gz"
shalla_file="${adb_tmpdir}/shallalist.txt"
curl ${curl_parm} --max-time "${adb_maxtime}" "${adb_arc_shalla}" --output "${shalla_archive}" 2>/dev/null
rc=${?}
if [ $((rc)) -eq 0 ]
if [ $((rc)) -ne 0 ]
then
f_log "shallalist archive download finished"
else
f_log "shallalist archive download failed (${adb_arc_shalla})" "${rc}"
f_restore
fi
@ -91,15 +90,10 @@ then
> "${shalla_file}"
for category in ${adb_cat_shalla}
do
tar -C "${adb_tmpdir}" -xzf "${shalla_archive}" BL/${category}/domains 2>/dev/null
tar -xOzf "${shalla_archive}" BL/${category}/domains 2>/dev/null >> "${shalla_file}"
rc=${?}
if [ $((rc)) -eq 0 ]
if [ $((rc)) -ne 0 ]
then
if [ -r "${adb_tmpdir}/BL/${category}/domains" ]
then
cat "${adb_tmpdir}/BL/${category}/domains" 2>/dev/null >> "${shalla_file}"
fi
else
f_log "shallalist archive extraction failed (${category})" "${rc}"
f_restore
fi
@ -157,9 +151,15 @@ done
# remove whitelist domains, sort domains and make them unique
# and finally rewrite ad/abuse domain information to dnsmasq file
#
> "${adb_dnsfile}"
grep -vxf "${adb_whitelist}" < "${adb_tmpfile}" 2>/dev/null | sort -u 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null >> "${adb_dnsfile}"
rc=${?}
if [ -s "${adb_whitelist}" ]
then
grep -Fvxf "${adb_whitelist}" "${adb_tmpfile}" 2>/dev/null | sort -u 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
rc=${?}
else
sort -u "${adb_tmpfile}" 2>/dev/null | eval "${adb_dnsformat}" 2>/dev/null > "${adb_dnsfile}"
rc=${?}
fi
if [ $((rc)) -eq 0 ]
then
rm -f "${adb_tmpfile}" >/dev/null 2>&1


+ 1
- 1
net/adblock/files/adblock.blacklist View File

@ -1 +1 @@
bild.de

+ 0
- 1
net/adblock/files/adblock.conf View File

@ -63,7 +63,6 @@ config source "shalla"
option adb_arc_shalla "http://www.shallalist.de/Downloads/shallalist.tar.gz"
list adb_catlist "adv"
list adb_catlist "costtraps"
list adb_catlist "downloads"
list adb_catlist "spyware"
list adb_catlist "tracker"
list adb_catlist "warez"

+ 1
- 4
net/adblock/files/adblock.whitelist View File

@ -1,4 +1 @@
downloads.openwrt.org
ftp.de.debian.org
download.eclipse.org
dl.sourceforge.net

Loading…
Cancel
Save