hostsblock: cli utility/cronscript to automate hostsfile adblocking

ambsace · 2015-05-19 16:07:06

Sorry for the noise, but I see that setting dnscacher=manual seems to work, and that you're talking about removing the auto detection anyways, so I guess there's no real issue here. Thanks for this script by the way.

kozaki · 2015-10-04 21:55:51

Resinstalled hostsblock this morning on a machine (with kwakd, no DNS caching).
After what that system had network issues: probs loading pages (www.cyberciti.biz, bookmark synchronization using diigo, Google search).

A look at top showed DNS processes maxing out CPU usage at 100% (and some RAM), one per request to the above mentionned domains:

% top (threads ON)
 1388 kozaki     4   0  735,4m 252,1m 19,7 12,5   0:10.59 R DNS Resolver #9
  ...

Logs:

oct. 03 17:07:16 gwenael kwakd[407]: [warning] Error sending data to client.
oct. 03 17:07:16 gwenael kwakd[407]: [warning] Error shutting down client socket.

Looked like what happened for another user : « kwakd is messing with X's sockets? the issue seems to disappear if I disable kwakd. » https://bbs.archlinux.org/viewtopic.php?id=176233
But stopping kwakd gave no results.

Solved the issue like this:
1. shifted to hostsblock with DNS caching (dnsmasq): nope...
but showed me that dnsmasq still used old /etc/hosts; /etc/hosts.block was not found on the system.
2. manually started hostsblock.service. Newly generated hosts.block is 4-5 times smaller than previous, dating from 2 days: 161573 vs 756304 lines (note: never touched its source lists).
3. told dnsmaq to use the new hostsblock generated meta-list (/etc/dnsmasq.conf line 132):

# or if you want it to read another file, as well as /etc/hosts, use this.
addn-hosts=/etc/hosts.block

4. replaced /etc/hosts with /etc/hostsblock/hosts.head
5. restarted dnsmasq

Voilà, all is back to work.

Haven't yet figured out what caused the DNS issue but it was hostsblock related.
Please let hostsblock users know they have to add the hosts.block line in their DNS caching daemon conf (it was in hostsblock.conf before if I remember well).

BTW conf's default settings are all to use hostsblock straight without DNS caching but that one:

hostshead="0"   # DEFAULT ## why default?                                        
#hostshead="/etc/hostsblock/hosts.head" # If NOT USING DNS CACHING.

Kuddos for an otherwise greatly appreciated tool!

amaro · 2015-12-07 08:53:36

I am trying to install hostsblock and there is no hosts.block file in /etc. Tried pacaur and manually and there is no such file.

cd /home/vb/Downloads/hostsblock/
[vb@evohp hostsblock]$ sudo install -Dm755 hostsblock.sh /usr/sbin/hostsblock
[vb@evohp hostsblock]$ sudo install -Dm755 hostsblock-urlcheck.sh /usr/sbin/hostsblock-urlcheck
[vb@evohp hostsblock]$ sudo install -Dm644 rc.conf /etc/hostsblock/rc.conf
install: cannot stat ‘rc.conf’: No such file or directory
[vb@evohp hostsblock]$ sudo install -Dm644 hostsblock.conf /etc/hostsblock/hostsblock.conf
[vb@evohp hostsblock]$ sudo install -Dm644 install -Dm644 black.list /etc/hostsblock/black.list
install: target ‘/etc/hostsblock/black.list’ is not a directory
[vb@evohp hostsblock]$ sudo install -Dm644 black.list /etc/hostsblock/black.list
[vb@evohp hostsblock]$ sudo install -Dm644 black.list /etc/hostsblock/white.list
[vb@evohp hostsblock]$ sudo install -Dm644 hosts.head /etc/hostsblock/hosts.head
[vb@evohp hostsblock]$ sudo hostsblock
[WARN] FAILED to refresh/download blocklist http://www.malwaredomainlist.com/hostslist/hosts.txt.
sort: cannot read: /etc/hosts.block: No such file or directory
cp: cannot stat ‘/etc/hosts.block’: No such file or directory
[WARN] FAILED to backup /etc/hosts.block.
[WARN] Postprocessing FAILED.

I am using dnsmasq and here are my settings of hostsblock.conf and dnsmasq.conf

# CACHE DIRECTORY. Directory where blocklists will be downloaded and stored.
# They will only be updated if the server has a newer version.
cachedir="/var/cache/hostsblock" # DEFAULT

# WORK DIRECTORY. Temporary directory where interim files will be unzipped and
# processed. This directory will be deleted after hostsblock completes itself.
tmpdir="/dev/shm" # DEFAULT

# FINAL HOSTSFILE. The final hosts file that combines together all downloaded
# blocklists. If not using a dns caching daemon like dnsmasq, this should be
# /etc/hosts. Include hosts file entries which you want to maintain in the
# "hostshead" entry, such as your loopback device (e.g. 127.0.0.1 localhosts)
hostsfile="/etc/hosts.block" # For use with a dnscacher, e.g dnsmasq
#hostsfile="/etc/hosts" # DEFAULT. If not using a dns caching daemon

# REDIRECT URL. Url to which blocked hosts will be redirect, either 0.0.0.0 or
# 127.0.0.1. This replaces any entries to 0.0.0.0 and 127.0.0.1. If you run a
# pixelserver such as pixelserv or kwakd, it is advisable to use 127.0.0.1.
redirecturl="127.0.0.1" # DEFAULT

# ALLOW REDIRECTS. Give "1" here to allow entries from blocklists that redirect
# rather than just block. Redirects present a certain security hazard, so make
# sure that you trust your blocklist providers if you give "1" here. Redirects
# allow those who have corrupted DNS servers to access sites not available
# otherwise.
redirects="0" # DEFAULT

# POSTPROCESSING SUBROUTINE. Commands to run after the new hostsfile has been
# created. You may configure your dnscacher here (see example), or use the new
# automatic detection routine.
postprocess(){                        #
    systemctl restart dnsmasq.service # For dnsmasq under systemd
}                                     #

#postprocess(){                        #
#    /bin/true                         # DEFAULT. If not using a dns caching daemon
#}                                     #

# HEAD FILE. File containing hosts file entries which you want at the beginning
# of the resultant hosts file, e.g. for loopback devices and IPV6 entries. Use
# your original /etc/hosts file here if you are writing your final blocklist to
# /etc/hosts so as to preserve your loopback devices. Give hostshead="0" to
# disable this feature. For those targeting /etc/hosts, it is advisable to copy
# their old /etc/hosts file to this file so as to preserve existing entries.
#hostshead="0"
hostshead="/etc/hostsblock/hosts.head" # DEFAULT

# BLACKLISTED SUBDOMAINS. File containing specific subdomains to blacklist which
# may not be in the downloaded blacklists. Be sure to provide not just the
# domain, e.g. "google.com", but also the specific subdomain a la
# "adwords.google.com" without quotations.
blacklist="/etc/hostsblock/black.list"

# WHITELIST. File containing the specific subdomains to allow through that may
# be blocked by the downloaded blocklists. In this file, put a space in front of
# a string in order to let through that specific site (without quotations), e.g.
# " www.example.com" will unblock "http://www.example.com" but not
# "http://subdomain.example.com". Leave no space in front of the entry to
# unblock all subdomains that contain that string, e.g. ".dropbox.com" will let
# through "www.dropbox.com", "dl.www.dropbox.com", "foo.dropbox.com",
# "bar.dropbox.com", etc.
whitelist="/etc/hostsblock/white.list"

#### NEW PARAMETERS ####

# CONNECT_TIMEOUT. Parameter passed to curl. Determines how long to try to
# connect to each blocklist url before giving up.
connect_timeout=60 # DEFAULT

# RETRY. Parameter passed to curl. Number of times to retry connecting to
# each blocklist url before giving up.
retry=0 # DEFAULT

# BACKUP_OLD. Save old hosts.block file. Passing "0" or "no" here has hostsblock
# NOT backup the old hosts.block file. Passing "1" or "yes" here has hostsblock
# save an uncompressed copy of hosts.block. Passing a compression command with
# gzip-compatible syntax has hostsblock save a compressed copy of hosts.block.
# By default, hostsblock attempts to save a gzip-compressed copy. If it can't
# find gzip, it makes no backup.
# backup_old="gzip" # DEFAULT

# RECYCLE_OLD. If, for some reason, new downloaded blocklists contain fewer
# entries than previous versions, this option allows hostsblock to recycle the
# old main blocklist (e.g. hosts.block) so that old entries are preserved. In
# effect, the old hosts.block file becomes another downloaded blocklist.
# 1 = hostsblock WILL recycle old hosts.block, 0 = hostsblock will NOT.
recycle_old=1 # DEFAULT

# VERBOSITY. How much info hostsblock should provide while running. Follows
# the following scheme:
### Level 0: Only fatal errors
### Level 1: Level 0 + non-fatal errors
### Level 2: Level 1 + updates to cache files
### Level 3: Level 2 + narration of each major phase
### Level 4: Level 3 + step-by-step details of all processes
### Level 5: Level 4 + stdout/stderr from sub-processes like curl, zip, 7za, etc.
verbosity=1 # DEFAULT

# ANNOTATE. File where hostsblock will save information on which blocklist files
# affect which urls. Needed for hostsblock-urlcheck.
annotate=/var/lib/hostsblock.db.gz

# BLOCKLISTS. List all blocklists to be downloaded as an array.Files may be
# either plaintext, zip, or 7z files. Hostsfiles will identify zip and 7z files
# by their file extension.

blocklists=(
## HIGHLY RECOMMENDED LISTS
'http://winhelp2002.mvps.org/hosts.zip'			# anti-ad and anti-malware list (winhelp2002.mvps.org/hosts.htm)
'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext' # anti-ad (pgl.yoyo.org/adservers/)
'http://hosts-file.net/download/hosts.zip'		# hpHosts main anti-ad and anti-malware (hosts-file.net/)
'http://www.malwaredomainlist.com/hostslist/hosts.txt'  # anti-malware (www.malwaredomainlist.com/)
'http://hosts-file.net/ad_servers.txt'			# anti-ads-only version of hpHosts main list (hosts-file.net)
## RECOMMENDED LISTS
'http://hosts-file.net/hphosts-partial.asp'		# hpHosts inter-release (hosts-file.net/)
'http://hostsfile.org/Downloads/BadHosts.unx.zip'	# anti-malware (hostsfile.org/hosts.html)
'http://hostsfile.mine.nu/Hosts.zip'			# anti-ad (hostsfile.mine.nu)
'http://someonewhocares.org/hosts/hosts'		# anti-ad and anti-malware (someonewhocares.org/hosts/)
'http://sysctl.org/cameleon/hosts'			# anti-ad (sysctl.org/cameleon/)
## OPTIONAL LISTS
#'http://hosts-file.net/download/yahoo_servers.zip'     # hpHosts for yahoo adservers (hosts-file.net/)
#'http://adblock.gjtech.net/?format=unix-hosts'         # another general-use list to block adds
#'http://adblock.mahakala.is/hosts'                     # more anti-ad
#'http://optimate.dl.sourceforge.net/project/adzhosts/HOSTS.txt' # anti-ad specifically for smartphones
#'https://hosts.neocities.org/'                         # anti-ad metalist
#'http://abp.mozilla-hispano.org/nauscopio/hosts.zip'	# supplemented version of someonewhocares.org list (https://nauscopio.wordpress.com/category/filtrado/)
#'http://rlwpx.free.fr/WPFF/htrc.7z'			# anti-trackers (very large) (rlwpx.free.fr/WPFF)
#'http://rlwpx.free.fr/WPFF/hpub.7z'			# anti-advertising (very large) (rlwpx.free.fr/WPFF)
#'http://rlwpx.free.fr/WPFF/hrsk.7z'			# anti-malware (very large) (rlwpx.free.fr/WPFF)
#'http://rlwpx.free.fr/WPFF/hsex.7z'			# anti-porn (very large) (rlwpx.free.fr/WPFF)
#'http://rlwpx.free.fr/WPFF/hmis.7z'			# miscellaneous sites to be blocked (rlwpx.free.fr/WPFF)
## LISTS WITH REDIRECTIONS. Enable redirects="1" to enable, and also uncomment
#'http://rlwpx.free.fr/WPFF/hblc.7z'			# redirections for phished sites (rlwpx.free.fr/WPFF)
)

listen-address=127.0.0.1,192.168.1.1
addn-hosts=/etc/hosts.block

Last edited by amaro (2015-12-07 08:54:23)

amaro · 2015-12-07 09:28:15

Found similar situation in the thread https://bbs.archlinux.org/viewtopic.php … 1#p1387261. So let's check

cat /etc/dnsmasq.conf | grep -v \# | tr -s '\n'
listen-address=127.0.0.1
addn-hosts=/etc/hosts.block

less /etc/hosts.block | wc
 443607  887216 14695330

grep giveawayoftheday /etc/hosts.block
127.0.0.1 giveawayoftheday.com
127.0.0.1 www.giveawayoftheday.com

cat /etc/resolv.conf
# Generated by resolvconf
nameserver 127.0.0.1

cat /etc/resolv.conf.head 
#dnsmasq
nameserver 127.0.0.1

What am I missing?

p.s. Putting 'resolv-file=/etc/hosts.block' in /etc/dnsmasq.conf as suggested here https://bbs.archlinux.org/viewtopic.php … 5#p1388115 gives me: Proxy server is refusing connection.

Last edited by amaro (2015-12-07 09:52:48)

amaro · 2015-12-07 09:38:51

Well

systemctl status hostsblock.service
● hostsblock.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

sudo systemctl start hostsblock.service
Failed to start hostsblock.service: Unit hostsblock.service failed to load: No such file or directory.

How did you do it?

kozaki wrote:

/etc/hosts.block was not found on the system.
2. manually started hostsblock.service.

amaro · 2015-12-07 10:30:21

Upon reboot I got this

sudo hostsblock
hostsblock-common.sh NOT FOUND. INSTALL IT TO /usr/lib/ OR /usr/local/lib/. EXITING...

sudo install -Dm644 hostsblock-common.sh /usr/lib/hostsblock-common.sh
sudo install -Dm644 hostsblock.service /usr/lib/systemd/system/hostsblock.service

sudo systemctl start hostsblock.service

systemctl status hostsblock.service
● hostsblock.service - Block bad domains system-wide
   Loaded: loaded (/usr/lib/systemd/system/hostsblock.service; static; vendor preset: disabled)
   Active: inactive (dead)

Dec 07 12:22:11 evohp systemd[1]: Starting Block bad domains system-wide...
Dec 07 12:22:37 evohp hostsblock[2398]: grep: write error
Dec 07 12:22:51 evohp systemd[1]: Started Block bad domains system-wide.
Dec 07 12:22:51 evohp hostsblock[2398]: [WARN] Postprocessing FAILED.

jasonwryan · 2015-12-07 15:49:17

Learn how to use the Edit button and stop bumping the thread...

ondoho · 2016-12-21 08:12:22

youtube not blocked but doesn't load

In all my browsers, I'm getting "Server not found" or "This site can’t be reached because server DNS address could not be found" for youtube.com.

youtube.com is not blocked in /etc/hosts, and neither are google.com or ytimg.com.
i also tried unblocking s.youtube.com and s2.youtube.com.

i can ping youtube.com just fine.

after I do

mv /etc/hosts /etc/hosts~ && cp /etc/hostsblock/hosts.head /etc/hosts

youtube.com loads (even with all my browser addons, e.g. noscript, active).

how can I troubleshoot this?
nothing in /etc/hosts or /etc/hostsblock/* seems to hint at youtube being blocked.
i tried the network tab of palemoon's and brave's developer tools, but the only network request goes out to youtube.com itself.
what is youtube looking for?

0strodamus · 2016-12-22 00:54:06

Maybe wireshark? Another thing that has tripped me up in the past is to make sure if you're using hostsblock-urlcheck to search, try not only youtube.com, but also www.youtube.com, etc.

ondoho · 2016-12-22 06:53:15

i was actually grepping /etc/hosts itself for youtube.com.

i will look into wireshark, thanks for that.

Last edited by ondoho (2016-12-22 06:53:55)

Docbroke · 2016-12-22 09:25:57

amaro wrote:

I am trying to install hostsblock and there is no hosts.block file in /etc. Tried pacaur and manually and there is no such file.

just copy your current /etc/hosts to /etc/hosts.block, or create new blank file, it will be overwritten by hostsblock

Last edited by Docbroke (2016-12-22 09:27:02)

ondoho · 2016-12-23 07:32:46

ondoho wrote:

youtube not blocked but doesn't load

i used tcpdump - this one needs to be unblocked:

youtube-ui.l.google.com

https://github.com/gaenserich/hostsblock/issues/65

setone · 2017-04-07 20:08:06

blacklist entries are not always handled correctly

If I add an entry to my black.list via hostsblock-urlcheck, it will not get added to hosts.block if there is an existing entry with a more restricted domain. For example, if I'm already blocking very.bad.site.com, and I add bad.site.com to the blacklist, it will not show up in hosts.block, and my DNS server (after restart of course) will forward the query and resolve bad.site.com.

I can take care of this problem with a tweak in the hostsblock script:

202c202
<         grep -q "$_blacklistline" "$hostsfile" || echo "$redirecturl $_blacklistline" >> "$hostsfile"
---
>         grep -qx "$_blacklistline" "$hostsfile" || echo "$redirecturl $_blacklistline" >> "$hostsfile"

There is a warning in hostsblock.conf, to the effect that additions to the blacklist should be more specific than the most basic domain names, but note that many of the entries coming from the block lists are indeed very stripped-down domains.

Last edited by setone (2017-04-08 04:33:11)

Unia · 2017-04-08 08:38:20

^ I reported this issue at the issue tracker on GitHub: https://github.com/gaenserich/hostsblock/issues/66

setone · 2017-04-08 16:16:56

Excellent, thank you Unia

rv546 · 2017-04-26 14:07:25

My hostsblock adds double entries to my hosts file that only differ in whitespace, nearly doubling the filesize:

.....
127.0.0.1 0000a-fast-proxy.de
127.0.0.1 0000a-fast-proxy.de
127.0.0.1 0000pv6.rxportalhosting.com
127.0.0.1 0000pv6.rxportalhosting.com
.....

How can i filter these?

Ubuntu 14.04, no dnsmasq

Alad · 2017-04-26 14:22:37

These forums are for Arch support only. Please redirect your queries to the author's github if you use his script on some different distribution.

https://github.com/gaenserich/hostsblock

Arch Linux

#301 2015-05-19 16:07:06

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#302 2015-10-04 21:55:51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#303 2015-12-07 08:53:36

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#304 2015-12-07 09:28:15

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#305 2015-12-07 09:38:51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#306 2015-12-07 10:30:21

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#307 2015-12-07 15:49:17

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#308 2016-12-21 08:12:22

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#309 2016-12-22 00:54:06

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#310 2016-12-22 06:53:15

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#311 2016-12-22 09:25:57

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#312 2016-12-23 07:32:46

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#313 2017-04-07 20:08:06

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#314 2017-04-08 08:38:20

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#315 2017-04-08 16:16:56

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#316 2017-04-26 14:07:25

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#317 2017-04-26 14:22:37

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Board footer