hostsblock: cli utility/cronscript to automate hostsfile adblocking

WonderWoofy · 2014-02-16 22:06:23

@Unia, hostsblock is quite a bit more than hosts-update. It has a whitelist and blacklist and a configuration file that allows you to selectively enable/disable different sources in list creation. It appears to also have a basic ability to check for updates by checking to see if the sources have changed. Oh, and it allows you to set where you want the list to be created, which allows you to keep a clean /etc/hosts and use something like dnsmasq to load the generated file.

Unia · 2014-02-16 22:07:53

I see! I will go for hostsblock, thanks!

gaenserich · 2014-02-19 00:15:10

Unia wrote:

I am pretty sure that this has been asked before, but at the moment my searching skills are not good enough to come up with any results. I was wondering what the differences are between hostsblock and hosts-update? I'm very unfamiliar with all this stuff so please bear with me here. The way I see it, hostsblock has the same functionality as hosts-update but a little more to it, such as the hostsblock-urlcheck script.
Am I right in assuming this?

Here are a few of the differences:

--pro hosts-update--
1) hosts-update looks a bit more polished in documentation (it has a man page) and installation procedure (uses the standard make procedure)
2) hosts-update appears to be more cross-platforum, or at least more easily cross-platforum. You can (and I have) run hostsblock on OS X, although there are certain limitations in OS X's use of the /etc/hosts file that discourage its heavy use as add blocker.
3) graysky appears to be a bit more active in maintaining his script. I would put more effort in, but I'm a stay-home dad working on a dissertation, which inhibits how much time I can spend on hostsblock.
4) hosts-update is a good bit simpler than hostsblock, making it a bit easier to hack to your liking.

--pro hostsblock--
a) hostsblock supports multiple blocklists using a variety of compression methods (plaintext, zip, 7zip), while hosts-update only supports using one specific blocklist.
b) because of (a), hostsblock will create a ridiculously huge /etc/hosts file, which means it blocks a lot more.
c) hostsblock allows you much more flexibility. You can use dnsmasq with hostsblock (to allow more control and possibly speed over just using /etc/hosts straight), use 127.0.0.1 or 0.0.0.0 as redirect ip addresses, etc.
d) hostsblock is more "bandwidth efficient". Instead of downloading its blocklists anew every time it runs (as hosts-update does), it instead keeps local cached copies and only downloads new copies if it detects updates to the blocklist(s).
e) more advanced whitelist and blacklist functions. You can specify (through hostblock-urlcheck for instance) additional url domains that you want blocked or unblocked.
f) hostsblock-urlcheck, as you already know, which is a (hacked together) script to help unblock or block domain urls found at a given address.

Feel free to take a peek at hostblock's github page, and, if you like it, feel free to install it and hack it to your liking...I'm always eager to get feedback and/or patches!

Unia · 2014-02-19 13:03:58

^ Thanks for the elaboration! I have already started using hostsblock and everything works fine!

One more question though; how often would you advise to update the blocklists?

gaenserich · 2014-02-19 17:07:03

I have it set up to run once a day via systemd, although that could be a bit overkill: It does take a lot of cpu time to sort and format all of those blocklists into a single file. If hostsblocks detects that just one entry in one of the blocklists has changed, it has to process (but download) all of the files over again.

Unia · 2014-02-19 17:08:06

I guess I'll do it whenever I remember I should

thiagowfx · 2014-03-01 00:22:58

My setup doesn't seem to be working. I use NetworkManager with dnsmasq,

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=keyfile
dns=dnsmasq

Because of this, my dnsmasq systemd unit is disabled (as the wiki recommends)
This is my /etc/dnsmasq.conf file, without the comments and the blank lines:

$ cat /etc/dnsmasq.conf | grep -v \# | tr -s '\n'

listen-address=127.0.0.1
addn-hosts=/etc/hosts.block

I've already runned the hostsblock executable, so my /etc/hosts.block file is full of web addresses (one easy example: eztv.it), no doubt:

$ less /etc/hosts.block | wc
 450522  901046 16421961

$ grep eztv.it /etc/hosts.block
127.0.0.1 eztv.it
127.0.0.1 www.eztv.it

Finally, as expected:

$ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 127.0.0.1

But I can access eztv.it from both Chromium and ping. So I guess the setup is not working as intended. Any clues?

Last edited by thiagowfx (2014-03-01 21:33:05)

ooo · 2014-03-01 00:58:40

^I use NetworkManager as well, and for me it works fine without the dnsmasq-line in NetworkManager.conf, and enabling the systemd service.

thiagowfx · 2014-03-01 01:33:25

@ooo I switched my configs to match yours. Initially I lost my connectivity but, later, after disconnecting then reconnecting again from my Home Network with nm-applet, I recovered it. But my /etc/resolv.conf file changed, now it contains three DNS servers (probably taken from my router).

I can still ping eztv.it (as any other site from the blocklist). Do you have an additional setup in your configs?

ooo · 2014-03-01 13:32:35

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry:

$ cat /etc/resolv.conf.head 
#dnsmasq
nameserver 127.0.0.1

lahwaacz · 2014-03-01 13:41:56

ooo wrote:

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry:
$ cat /etc/resolv.conf.head 
#dnsmasq
nameserver 127.0.0.1

AFAIK, /etc/resolv.conf.head is used only by dhcpcd (i.e. NetworkManager does not use it directly). See https://wiki.archlinux.org/index.php/Ne … esolv.conf

thiagowfx · 2014-03-01 21:30:59

ooo wrote:

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry

lahwaacz wrote:

AFAIK, /etc/resolv.conf.head is used only by dhcpcd (i.e. NetworkManager does not use it directly).

Nice @ooo, this worked like a charm, the localhost IP gets prepended to my router's three DNS servers.

@lahwaacz, I investigated and found the following: network-manager depends on dhcp-client; both dhcpcd and dhclient (the last one pointed to on the wiki) provide it. In my case I am using dhcpcd, so @ooo's approach worked out-of-the-box. Also, notice that dhcpcd is on the base group, so it will probably be the default for most users.

I'll just add the following: after a proper reboot, ping worked OK on the blocked sites, but I could still access them from Chromium. After cleaning up everything (Ctrl+Shift+Del shortcut) and restarting Chromium, the websites got blocked (great!). One thing is strange, though: if I access eztv.it (oh man, I'll whitelist it soon) from the Incognito mode, I get a blank page from the kwakd daemon, like I would expect. Buuuuut if I access it outside Incognito, I get an error page (you know, like "This webpage is not available").

On Midori, everything works as expected (both into Private mode browsing and outside it).

So I guess Chromium is a bit problematic (trying to get into your way) about caching / pre-fetching websites. See [1], [2] and [3] below.

[1]: https://stackoverflow.com/questions/152 … -in-chrome
[2]: https://superuser.com/questions/648133/ … ts-on-os-x
[3]: http://fatbloke.tumblr.com/post/7228329 … ng-dnsmasq

Thanks, people.

Last edited by thiagowfx (2014-03-01 21:34:00)

thiagowfx · 2014-03-02 20:56:02

I've just discovered the problem with Chromium. It is related to DNS prefetching. I'll leave here a link with the solution: https://superuser.com/questions/723703/ … nd-dnsmasq.

moist · 2014-03-03 11:35:22

I'm having difficulty getting this to work. I'm using dhcpcd and dnsmasq/kwakd are running, I installed hostsblock through the AUR.
Running hostsblock gives me no output, but /etc/hosts.block seems to have been populated.
Hostsblock-urlcheck does provide output and lets me block/unblock, but I'm still getting served every single ad through surf from what I can tell.

Last edited by moist (2014-03-03 12:39:30)

WonderWoofy · 2014-03-03 16:16:21

@moist, it sounds as though your /etc/resolv.conf is still pointing its 'naemserver' lines at an external DNS. You need to have this pointing to 127.0.0.1 (or really 127.anything, since anything starting with 127 points back to the localhost) so that it uses dnsmasq.

To do this, you need to use some mechanism to have 'nameserver 127.0.0.1' prepended to your /etc/resolv.conf. The wiki has some suggestions, but since you are using dhcpcd, you can simply put this one line in /etc/resolv.conf.head and it will be stuck on the top of that file for you.

lahwaacz · 2014-03-03 16:21:27

@moist: You will most likely need to set 'resolv-file=/etc/hosts.block' in /etc/dnsmasq.conf. Also read the comment in /etc/hostsblock/rc.conf regarding the 'hostsfile' option.

thiagowfx · 2014-03-03 16:28:35

moist wrote:

Running hostsblock gives me no output

This is normal, you can look for logs in /var/log/hostsblock.log (by default, unless you've changed the location)

Last edited by thiagowfx (2014-03-03 16:29:41)

moist · 2014-03-03 18:32:47

Thanks for the hints WonderWoofy, lahwaacz and thiagowfx. Managed to get it working.

midfingr · 2014-03-20 02:15:20

Hello all,

First of all. A big thank you! to 'gaenserich' and the folks for this great tool.

Everything pretty much works as expected, with a couple of exceptions:

1. chromium-pepper-flash (stable); boilerplate page with kwakd shows 'The server refused the connection.'
see image - http://i.imgur.com/S64zxGD.jpg

2. block Facebook sponsored ads?
image - http://i.imgur.com/LB4S3I5.jpg

*Chrome AdBlock removes these in both instances FTR.

Any thoughts would be greatly appreciated.

Thanks,
'Mid'

ondoho · 2014-03-21 05:53:54

@midfingr:

hostsblock downloads hosts files others have compiled and installs them to your computer (without duplicate entries).
it does not do the actual blocking - that is part of the operating system, without going into detail.
take a look at the config files in /etc/hostsblock (and add/remove links to hosts lists) and use the hostsblock-urlcheck script to add/remove particular urls.

i think your questions are valid and i'm sure we'd all love hostsblock to be a 100% adblock replacement.
but keep in mind that you're getting url-blocking with minimal overhead and no additional software running on your machine/browser.

ooo · 2014-03-21 14:13:15

midfingr wrote:

1. chromium-pepper-flash (stable); boilerplate page with kwakd shows 'The server refused the connection.'
2. block Facebook sponsored ads?

1. sounds like kwakd isn't running/working correctly

2. hostsblock blocks all content from certain servers by redirecting them to localhost (or rather the OS does it, hostsblock only grabs the ad server lists from internet). facebook and google ads & such are 'embedded' inside the html code so you can't block them by simply blocking some server.

In my experience using adblock plus in chromium with hostsblock is still faster than only using adblock plus, while still blocking all the ads. but ymmv
you can also disable adblock for pages that are already ad-free with hostsblock only

F34R · 2014-05-25 19:40:59

Whaoo i found it. my /etc/hosts list alreay managed or added easily more sites ? So is replaced the bloated adblock and how to convert blocked list to host ?

ontobelli · 2014-06-02 07:33:56

Hello @gaenserich

Just reporting a "bug"

Since last update from http://rlwpx.free.fr/WPFF/ they removed the prefix 127.0.0.1 from each line and hostsblock srcript fails to proccess them correcty,

Thank you!

EDIT:

As a workarround in case anybody need it.

I edit *manually* those list and included the hosts in the file "black.list" so they are included in the "hosts.block".

EDIT 2:
They fixed the files. Everithing is ok now.

Last edited by ontobelli (2014-06-07 16:08:18)

sadi · 2014-06-30 09:15:33

Recently I've begun encountering a problem: hostsblock tries to connect a server (pgl.yoyo.org) but the connection fails, which results in hostsblock aborting the cron job.

Could hostsblock perhaps use the latest cache file in such cases, perhaps after a couple of retries?

gaenserich · 2014-07-20 15:40:50

Hi all...sorry for the long absence...I have been working on a big re-write of hostsblock for the bast month that addresses a few of the above issues (e.g. using the cachefile even when download fails). I've stalled out a bit on it as of late (baby got sick, and then gave the sickness to me), but it is available for a pull at github https://github.com/gaenserich/hostsblock

Arch Linux

#251 2014-02-16 22:06:23

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#252 2014-02-16 22:07:53

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#253 2014-02-19 00:15:10

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#254 2014-02-19 13:03:58

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#255 2014-02-19 17:07:03

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#256 2014-02-19 17:08:06

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#257 2014-03-01 00:22:58

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#258 2014-03-01 00:58:40

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#259 2014-03-01 01:33:25

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#260 2014-03-01 13:32:35

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#261 2014-03-01 13:41:56

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#262 2014-03-01 21:30:59

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#263 2014-03-02 20:56:02

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#264 2014-03-03 11:35:22

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#265 2014-03-03 16:16:21

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#266 2014-03-03 16:21:27

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#267 2014-03-03 16:28:35

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#268 2014-03-03 18:32:47

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#269 2014-03-20 02:15:20

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#270 2014-03-21 05:53:54

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#271 2014-03-21 14:13:15

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#272 2014-05-25 19:40:59

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#273 2014-06-02 07:33:56

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#274 2014-06-30 09:15:33

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#275 2014-07-20 15:40:50

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Board footer