You are not logged in.

#251 2014-02-16 22:06:23

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@Unia, hostsblock is quite a bit more than hosts-update.  It has a whitelist and blacklist and a configuration file that allows you to selectively enable/disable different sources in list creation.  It appears to also have a basic ability to check for updates by checking to see if the sources have changed.  Oh, and it allows you to set where you want the list to be created, which allows you to keep a clean /etc/hosts and use something like dnsmasq to load the generated file.

Offline

#252 2014-02-16 22:07:53

Unia
Member
From: The Netherlands
Registered: 2010-03-30
Posts: 2,471
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I see! I will go for hostsblock, thanks!


If you can't sit by a cozy fire with your code in hand enjoying its simplicity and clarity, it needs more work. --Carlos Torres

Offline

#253 2014-02-19 00:15:10

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Unia wrote:

I am pretty sure that this has been asked before, but at the moment my searching skills are not good enough to come up with any results. I was wondering what the differences are between hostsblock and hosts-update? I'm very unfamiliar with all this stuff so please bear with me here. The way I see it, hostsblock has the same functionality as hosts-update but a little more to it, such as the hostsblock-urlcheck script.

Am I right in assuming this?

Here are a few of the differences:

--pro hosts-update--
1) hosts-update looks a bit more polished in documentation (it has a man page) and installation procedure (uses the standard make procedure)
2) hosts-update appears to be more cross-platforum, or at least more easily cross-platforum. You can (and I have) run hostsblock on OS X, although there are certain limitations in OS X's use of the /etc/hosts file that discourage its heavy use as add blocker.
3) graysky appears to be a bit more active in maintaining his script. I would put more effort in, but I'm a stay-home dad working on a dissertation, which inhibits how much time I can spend on hostsblock.
4) hosts-update is a good bit simpler than hostsblock, making it a bit easier to hack to your liking.

--pro hostsblock--
a) hostsblock supports multiple blocklists using a variety of compression methods (plaintext, zip, 7zip), while hosts-update only supports using one specific blocklist.
b) because of (a), hostsblock will create a ridiculously huge /etc/hosts file, which means it blocks a lot more.
c) hostsblock allows you much more flexibility. You can use dnsmasq with hostsblock (to allow more control and possibly speed over just using /etc/hosts straight), use 127.0.0.1 or 0.0.0.0 as redirect ip addresses, etc.
d) hostsblock is more "bandwidth efficient". Instead of downloading its blocklists anew every time it runs (as hosts-update does), it instead keeps local cached copies and only downloads new copies if it detects updates to the blocklist(s).
e) more advanced whitelist and blacklist functions. You can specify (through hostblock-urlcheck for instance) additional url domains that you want blocked or unblocked.
f) hostsblock-urlcheck, as you already know, which is a (hacked together) script to help unblock or block domain urls found at a given address.

Feel free to take a peek at hostblock's github page, and, if you like it, feel free to install it and hack it to your liking...I'm always eager to get feedback and/or patches!


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#254 2014-02-19 13:03:58

Unia
Member
From: The Netherlands
Registered: 2010-03-30
Posts: 2,471
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

^ Thanks for the elaboration! I have already started using hostsblock and everything works fine!

One more question though; how often would you advise to update the blocklists?


If you can't sit by a cozy fire with your code in hand enjoying its simplicity and clarity, it needs more work. --Carlos Torres

Offline

#255 2014-02-19 17:07:03

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I have it set up to run once a day via systemd, although that could be a bit overkill: It does take a lot of cpu time to sort and format all of those blocklists into a single file. If hostsblocks detects that just one entry in one of the blocklists has changed, it has to process (but download) all of the files over again.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#256 2014-02-19 17:08:06

Unia
Member
From: The Netherlands
Registered: 2010-03-30
Posts: 2,471
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I guess I'll do it whenever I remember I should tongue


If you can't sit by a cozy fire with your code in hand enjoying its simplicity and clarity, it needs more work. --Carlos Torres

Offline

#257 2014-03-01 00:22:58

thiagowfx
Member
From: Los Angeles
Registered: 2013-07-09
Posts: 532
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

My setup doesn't seem to be working. I use NetworkManager with dnsmasq,

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=keyfile
dns=dnsmasq

Because of this, my dnsmasq systemd unit is disabled (as the wiki recommends)
This is my /etc/dnsmasq.conf file, without the comments and the blank lines:

$ cat /etc/dnsmasq.conf | grep -v \# | tr -s '\n'

listen-address=127.0.0.1
addn-hosts=/etc/hosts.block

I've already runned the hostsblock executable, so my /etc/hosts.block file is full of web addresses (one easy example: eztv.it), no doubt:

$ less /etc/hosts.block | wc
 450522  901046 16421961
$ grep eztv.it /etc/hosts.block
127.0.0.1 eztv.it
127.0.0.1 www.eztv.it

Finally, as expected:

$ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 127.0.0.1

But I can access eztv.it from both Chromium and ping. So I guess the setup is not working as intended. Any clues?

Last edited by thiagowfx (2014-03-01 21:33:05)

Offline

#258 2014-03-01 00:58:40

ooo
Member
Registered: 2013-04-10
Posts: 956

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

^I use NetworkManager as well, and for me it works fine without the dnsmasq-line in NetworkManager.conf, and enabling the systemd service.

Offline

#259 2014-03-01 01:33:25

thiagowfx
Member
From: Los Angeles
Registered: 2013-07-09
Posts: 532
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@ooo I switched my configs to match yours. Initially I lost my connectivity but, later, after disconnecting then reconnecting again from my Home Network with nm-applet, I recovered it. But my /etc/resolv.conf file changed, now it contains three DNS servers (probably taken from my router).

I can still ping eztv.it (as any other site from the blocklist). Do you have an additional setup in your configs?

Offline

#260 2014-03-01 13:32:35

ooo
Member
Registered: 2013-04-10
Posts: 956

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry:

$ cat /etc/resolv.conf.head 
#dnsmasq
nameserver 127.0.0.1

Offline

#261 2014-03-01 13:41:56

lahwaacz
Wiki Admin
From: Czech Republic
Registered: 2012-05-29
Posts: 433

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

ooo wrote:

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry:

$ cat /etc/resolv.conf.head 
#dnsmasq
nameserver 127.0.0.1

AFAIK, /etc/resolv.conf.head is used only by dhcpcd (i.e. NetworkManager does not use it directly). See https://wiki.archlinux.org/index.php/Ne … esolv.conf

Offline

#262 2014-03-01 21:30:59

thiagowfx
Member
From: Los Angeles
Registered: 2013-07-09
Posts: 532
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

ooo wrote:

@thiagowfx, one more thing: since /etc/resolv.conf gets overwritten by NetworkManager you need /etc/resolv.conf.head with the dnsmasq entry

lahwaacz wrote:

AFAIK, /etc/resolv.conf.head is used only by dhcpcd (i.e. NetworkManager does not use it directly).

Nice @ooo, this worked like a charm, the localhost IP gets prepended to my router's three DNS servers.

@lahwaacz, I investigated and found the following: network-manager depends on dhcp-client; both dhcpcd and dhclient (the last one pointed to on the wiki) provide it. In my case I am using dhcpcd, so @ooo's approach worked out-of-the-box. Also, notice that dhcpcd is on the base group, so it will probably be the default for most users.


I'll just add the following: after a proper reboot, ping worked OK on the blocked sites, but I could still access them from Chromium. After cleaning up everything (Ctrl+Shift+Del shortcut) and restarting Chromium, the websites got blocked (great!). One thing is strange, though: if I access eztv.it (oh man, I'll whitelist it soon) from the Incognito mode, I get a blank page from the kwakd daemon, like I would expect. Buuuuut if I access it outside Incognito, I get an error page (you know, like "This webpage is not available").

On Midori, everything works as expected (both into Private mode browsing and outside it).

So I guess Chromium is a bit problematic (trying to get into your way) about caching / pre-fetching websites. See [1], [2] and [3] below.

[1]: https://stackoverflow.com/questions/152 … -in-chrome
[2]: https://superuser.com/questions/648133/ … ts-on-os-x
[3]: http://fatbloke.tumblr.com/post/7228329 … ng-dnsmasq

Thanks, people.

Last edited by thiagowfx (2014-03-01 21:34:00)

Offline

#263 2014-03-02 20:56:02

thiagowfx
Member
From: Los Angeles
Registered: 2013-07-09
Posts: 532
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I've just discovered the problem with Chromium. It is related to DNS prefetching. I'll leave here a link with the solution: https://superuser.com/questions/723703/ … nd-dnsmasq.

Offline

#264 2014-03-03 11:35:22

moist
Member
Registered: 2013-04-21
Posts: 12

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I'm having difficulty getting this to work. I'm using dhcpcd and dnsmasq/kwakd are running, I installed hostsblock through the AUR.
Running hostsblock gives me no output, but /etc/hosts.block seems to have been populated.
Hostsblock-urlcheck does provide output and lets me block/unblock, but I'm still getting served every single ad through surf from what I can tell.

Last edited by moist (2014-03-03 12:39:30)

Offline

#265 2014-03-03 16:16:21

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@moist, it sounds as though your /etc/resolv.conf is still pointing its 'naemserver' lines at an external DNS.  You need to have this pointing to 127.0.0.1 (or really 127.anything, since anything starting with 127 points back to the localhost) so that it uses dnsmasq. 


To do this, you need to use some mechanism to have 'nameserver 127.0.0.1' prepended to your /etc/resolv.conf.  The wiki has some suggestions, but since you are using dhcpcd, you can simply put this one line in /etc/resolv.conf.head and it will be stuck on the top of that file for you.

Offline

#266 2014-03-03 16:21:27

lahwaacz
Wiki Admin
From: Czech Republic
Registered: 2012-05-29
Posts: 433

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@moist: You will most likely need to set 'resolv-file=/etc/hosts.block' in /etc/dnsmasq.conf. Also read the comment in /etc/hostsblock/rc.conf regarding the 'hostsfile' option.

Offline

#267 2014-03-03 16:28:35

thiagowfx
Member
From: Los Angeles
Registered: 2013-07-09
Posts: 532
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

moist wrote:

Running hostsblock gives me no output

This is normal, you can look for logs in /var/log/hostsblock.log (by default, unless you've changed the location)

Last edited by thiagowfx (2014-03-03 16:29:41)

Offline

#268 2014-03-03 18:32:47

moist
Member
Registered: 2013-04-21
Posts: 12

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thanks for the hints WonderWoofy, lahwaacz and thiagowfx. Managed to get it working.

Offline

#269 2014-03-20 02:15:20

midfingr
Member
From: Canada
Registered: 2012-06-27
Posts: 26

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Hello all,

First of all. A big thank you! to 'gaenserich' and the folks for this great tool. smile

Everything pretty much works as expected, with a couple of exceptions:

1. chromium-pepper-flash (stable); boilerplate page with kwakd shows 'The server refused the connection.'
see image - http://i.imgur.com/S64zxGD.jpg

2. block Facebook sponsored ads?
image - http://i.imgur.com/LB4S3I5.jpg

*Chrome AdBlock removes these in both instances FTR.

Any thoughts would be greatly appreciated.

Thanks,
'Mid'

Offline

#270 2014-03-21 05:53:54

ondoho
Member
Registered: 2013-04-30
Posts: 302

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@midfingr:

hostsblock downloads hosts files others have compiled and installs them to your computer (without duplicate entries).
it does not do the actual blocking - that is part of the operating system, without going into detail.
take a look at the config files in /etc/hostsblock (and add/remove links to hosts lists) and use the hostsblock-urlcheck script to add/remove particular urls.

i think your questions are valid and i'm sure we'd all love hostsblock to be a 100% adblock replacement.
but keep in mind that you're getting url-blocking with minimal overhead and no additional software running on your machine/browser.

Offline

#271 2014-03-21 14:13:15

ooo
Member
Registered: 2013-04-10
Posts: 956

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

midfingr wrote:

1. chromium-pepper-flash (stable); boilerplate page with kwakd shows 'The server refused the connection.'

2. block Facebook sponsored ads?

1. sounds like kwakd isn't running/working correctly

2. hostsblock blocks all content from certain servers by redirecting them to localhost (or rather the OS does it, hostsblock only grabs the ad server lists from internet). facebook and google ads & such are 'embedded' inside the html code so you can't block them by simply blocking some server.

In my experience using adblock plus in chromium with hostsblock is still faster than only using adblock plus, while still blocking all the ads. but ymmv
you can also disable adblock for pages that are already ad-free with hostsblock only

Offline

#272 2014-05-25 19:40:59

F34R
Member
From: /dev/loliland
Registered: 2012-02-05
Posts: 238

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Whaoo i found it. my /etc/hosts list alreay managed or added easily more sites ? So is replaced the bloated adblock and how to convert blocked list to host ?

Offline

#273 2014-06-02 07:33:56

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Hello @gaenserich

Just reporting a "bug"

Since last update from http://rlwpx.free.fr/WPFF/ they removed the prefix 127.0.0.1 from each line and hostsblock srcript fails to proccess them correcty,

Thank you!


EDIT:

As a workarround in case anybody need it.

I edit *manually* those list and included the hosts in the file "black.list" so they are included in the "hosts.block".


EDIT 2:
They fixed the files. Everithing is ok now.

Last edited by ontobelli (2014-06-07 16:08:18)

Offline

#274 2014-06-30 09:15:33

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Recently I've begun encountering a problem: hostsblock tries to connect a server (pgl.yoyo.org) but the connection fails, which results in hostsblock aborting the cron job.

Could hostsblock  perhaps use the latest cache file in such cases, perhaps after a couple of retries?

Offline

#275 2014-07-20 15:40:50

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Hi all...sorry for the long absence...I have been working on a big re-write of hostsblock for the bast month that addresses a few of the above issues (e.g. using the cachefile even when download fails). I've stalled out a bit on it as of late (baby got sick, and then gave the sickness to me), but it is available for a pull at github https://github.com/gaenserich/hostsblock


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

Board footer

Powered by FluxBB