hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich · 2013-06-01 01:56:29

@ontobelli: What is your system locale? I have yet to have encountered an error using UTF-8 locales. As a super-quick fix, enable a UTF-8 locale and then add this to the beginning of /usr/bin/hostsblock to see if it helps: <code>LANG=de_DE.UTF-8</code> where "de_DE.UTF-8" is your enable utf locale.

Thanks all for the well-wishes with the baby...only 18 years to go before my time is my own again.

ontobelli · 2013-06-01 12:00:42

gaenserich wrote:

@ontobelli: What is your system locale?

/etc/locale.conf

LANG=en_US.UTF-8
LC_COLLATE=C

Ok. I'll try the quick-fix. Thanks.

EDIT:

Done and is working fine.

Last edited by ontobelli (2013-06-01 12:27:47)

ontobelli · 2013-06-01 13:25:43

Version 0.12.1 (31.05.2013) is not processing the white.list

gaenserich · 2013-06-02 15:39:36

@ontobelli: this should fix that...sorry for all the quick band-aid repairs...this is all I have time for at the moment.

Version 0.12.2 (02.06.2013)
*whitelist, blacklist not applying fixed

Doomcide · 2013-06-03 14:34:45

If it's bug-report time I'd like to contribute two bits:
1. With a light terminal scheme (black on white), the output of hostsblock-urlcheck is only partly readable. I think using the reset color escape code for the white/black parts would be cleaner solution.
2. I don't get output on my machine from running hostsblock, is this normal?

Btw.: I also made a crux-port:http://crux.nu/portdb/index.php?a=repo&q=doom. You can mention it on the website if you want
Also thank you very much for this, it's my method of adblocking since ~1 year.

Army · 2013-06-03 16:52:51

Doomcide wrote:

2. I don't get output on my machine from running hostsblock, is this normal?

Yes. You can however watch the status with

tail -f /var/log/hostsblock.log

Doomcide · 2013-06-03 18:00:30

^Ok, thank you. I've been wondering about this for a while.

gaenserich · 2013-06-03 20:33:15

@Army: Danke für den Tipp an Doomcide...Die Web-Präsenz deiner Band sieht auch echt ganz cool aus, insbesondere eure animierten Porträts. Welches Mitglied bist du denn?

@Doomcide: Could you point me to some sort of implementation or other example of what you would like with the urlcheck color codes? Even better: feel free to send me a patch or post a pull request to github.

Last edited by gaenserich (2013-06-04 01:26:54)

Doomcide · 2013-06-03 21:15:13

Here's a patch

--- hostsblock-urlcheck	2013-06-03 23:09:23.050914199 +0200
+++ hostsblock-urlcheck.modified	2013-06-03 23:10:18.437953606 +0200
@@ -24,7 +24,7 @@ fi
 # CHECK SUBROUTINE
 check(){
     if grep "[[:space:]]`echo $@ | sed 's|\.|\\\.|g'`$" "$hostsfile" &>/dev/null; then
-	printf "\e[1;31mBLOCKED: \e[0;37m'$@' \e[0;32mUnblock? \e[0;37m[y/N] "
+	printf "\e[1;31mBLOCKED: \e[0m'$@' \e[0;32mUnblock? \e[0m[y/N] "
         read a
         if [[ $a == "y" || $a == "Y" ]]; then
             echo "Unblocking $@"
@@ -34,7 +34,7 @@ check(){
             changed=1
         fi
     else
-        printf "\e[0;32mNOT BLOCKED: \e[0;37m'$@' \e[1;31mBlock? \e[0;37m[y/N] "
+        printf "\e[0;32mNOT BLOCKED: \e[0m'$@' \e[1;31mBlock? \e[0m[y/N] "
         read a
         if [[ $a == "y" || $a == "Y" ]]; then
             echo "Blocking $@"
@@ -70,4 +70,4 @@ else
         echo "Whole-page scan completed."
     fi
     [ "$changed" == "1" ] && postprocess &>/dev/null
-fi
\ No newline at end of file
+fi

gaenserich · 2013-06-03 21:33:24

Sold!:

Version 0.12.3 (03.06.2013)
*hostblock-urlcheck: issue with colored prompt on white-backgrounded terminals fixed

ontobelli · 2013-06-04 00:02:19

Suggestions:

1) Based on that block.list entries has a bigger probability of occurrence than the general list. Change the order of blacklist to be after the head instead of the end of file to save a few nanoseconds on each query.

(up)      # APPEND BLACKLIST ENTRIES 
(down)    # PROCESS AND WRITE TO FILE

2) Sometimes curl fails to retrieve a file and the hostsblock stops without any message to the user. Other times retrieve garbage or error messages from the http server and overwrites the file in /var/cache/hostsblock and continue processing the list and the new /etc/hosts is a subset of the/etc/hosts,old and user could became exposed to risks without notice.
In case of error I think is better to continue with the updating and processing of all lists and use the "last known good" cache of the non retried ones so the final /etc/hosts be "more complete". Maybe needed 2 copies of cache or a temporal file to avoid overwrite the "last known good" with a error message. Maybe needed a comparative of file sizes to detect a big reduction of size in the new file. Not sure if curl is able to detect that kind of errors. Just an idea.

gaenserich · 2013-06-04 01:25:50

@ontobelli: all very good ideas, but they undoubtedly need more attention than what I can spare at the moment. On my wish list is a bit of a re-write, perhaps in perl or python,, integration with systemd, better parallelization. In the end, I think that these ends would all be better served by a major revision and not by the little quick patches I've been pumping out.

Doomcide · 2013-06-04 09:29:14

@gaensrich: Thank you for accepting the patch. Even though these are just plans for some point in the future now, I'd like to ask to make systemd integration optional or in a way that it still works on non-systemd systems.

Edit: Another suggestion: Could you make tags for the versions on github? This would make packaging easier/cleaner.

Last edited by Doomcide (2013-06-04 11:03:10)

gaenserich · 2013-06-04 15:27:10

@doomcide: integration with systemd would most likely just be packaging a couple .service and even .timer files in addition to the script as we are already familiar with it. You will undoubtedly still be able to run it without systemd.

I'll have to check out how to make tags on github...I'm not entirely familiar with all the features of github yet.

WonderWoofy · 2013-06-04 16:41:16

gaenserich wrote:

@doomcide: integration with systemd would most likely just be packaging a couple .service and even .timer files in addition to the script as we are already familiar with it. You will undoubtedly still be able to run it without systemd.
I'll have to check out how to make tags on github...I'm not entirely familiar with all the features of github yet.

Maybe you should also make a one-liner script that the user can copy into /etc/cron.daily as well, just in case he/she wishes to use anacron instead. The timer units are cool, but it is a bit depenent on either being set to a time in which you know for certain that the computer will be on, or in the case of an actual timer, you are depending on the person's usage habits to not reboot very often. I really like the systemd.timer events, but as it is, we are just not at a point where it can replace the full functionality of anacron.

Here is a script that might work well in /etc/cron.weekly (or daily or whatever). I just took the logrotate script, and changed the command:

#!/bin/sh

# nicenesses range from -20 (most favorable scheduling) to 19 (least favorable)
NICE=19

# 0 for none, 1 for real time, 2 for best-effort, 3 for idle
IONICE_CLASS=2

# 0-7 (for IONICE_CLASS 1 and 2 only), 0=highest, 7=lowest
IONICE_PRIORITY=7

CMD_HOSTSBLOCK="/usr/bin/hostsblock"

if [ -x /usr/bin/nice ]; then
  CMD_HOSTSBLOCK="/usr/bin/nice -n ${NICE:-19} ${CMD_HOSTSBLOCK}"
fi

if [ -x /usr/bin/ionice ]; then
  CMD_HOSTSBLOCK="/usr/bin/ionice -c ${IONICE_CLASS:-2} -n ${IONICE_PRIORITY:-7} ${CMD_HOSTSBLOCK}"
fi

${CMD_HOSTSBLOCK}

exit 0

Doomcide · 2013-06-04 18:28:54

@gaensrich: Ok, thanks for the info. Take your time with tags, especially if you're busy at the moment.

ondoho · 2013-10-19 21:23:49

hello,
i took a few hours to set up hostsblock & dnsmasq.
no problems with hostsblock, but dnsmasq proved a bit tricky.
i got it working now, but i'm still not 100% sure i'm doing it properly.
please, have a look at my configurations:
/etc/hostsblock/rc.conf:

logfile="/var/log/hostsblock.log"
cachedir="/var/cache/hostsblock"
tmpdir="/dev/shm"
hostsfile="/etc/hosts.block"
redirecturl="127.0.0.1"
redirects="0"
postprocess(){                        #
    systemctl restart dnsmasq.service # For dnsmasq under systemd
}                                     #
hostshead="0"
blacklist="/etc/hostsblock/black.list"
whitelist="/etc/hostsblock/white.list"
blocklists=( ...

/etc/dnsmasq.conf:

# If you don't want dnsmasq to read /etc/resolv.conf or any other
# file, getting its servers from this file instead (see below), then
# uncomment this.
no-resolv <--------------------- important!
listen-address=127.0.0.1
addn-hosts=/etc/hosts.block

the marked entry (no-resolv) was commented out by default.
i had to uncomment it to make dnsmasq behave so that it actually blocks the ads from /etc/hosts.block.

i don't know, is it only me... i'm using mobile broadband, netctl starting a simple ppp connection. it seems that some things work differently than with a normal wired/less connection.

do you think i have it set up properly now? next i'd have to get kwakd working (right now it's complaining [panic] that it can't open the default port).
or try the 0.0.0.0 variant.

i'd like to point out a little glitch in this part of the documentation: the black block starting "postprocess()..." should be added to /etc/hostsblock/rc.conf, and NOT to /etc/dnsmasq.conf.
also i wouldn't rely solely on arch wiki for instructions on how to set up dnsmasq.

but, thanks to all for another great tool to reclaim control over my personal www. respect.

edit: i added "google.com" to the blacklist with hostsblock-urlchecker, and now the black.list looks like this:

adwords.google.comgoogle.com
www.google.com

:-(

Last edited by ondoho (2013-10-19 21:46:01)

WonderWoofy · 2013-10-19 23:56:31

There is no problem with having dnsmasq load /etc/resolv.conf. In fact, if you get your DNS server address via dhcp, it is pretty important as it will then know what servers it should actually be querying for domain name resolution. But you can alternatively use DNS servers set specifically in your dnsmasq.conf as well. If you remove the no-resolv from your config there, then run dnsmasq you can see in the journal that if you then connect to a network, overwriting /etc/resolv.conf, dnsmasq will pick up these new nameservers.

What you do need to do though is ensure that the first nameserver in /etc/resolv.conf is pointing to 127.0.0.1. There are a couple ways to do this depending on what tool you use to obtain your dhcp lease. So for the sake of an example, if you were using dhcpcd, you could use /etc/resolv.conf.head. Anything in that file will be applied to the resolv.conf first. So simply putting "nameserver 127.0.0.1" in there should work. I think though that networkmanager does not parse or apply this file, so a different method of setting this is required.

In my config, I have the following:

% sed -e '/^#/d' -e '/^[ ]*$/d' /etc/resolv.conf
server=8.8.8.8
server=8.8.4.4
listen-address=127.0.0.1
addn-hosts=/etc/hosts.block
cache-size=1000
all-servers

In my journal, I see this:

Oct 19 11:18:06 thinkpad dnsmasq[296]: read /etc/hosts.block - 205890 addresses
Oct 19 11:18:15 thinkpad dnsmasq[296]: reading /etc/resolv.conf
Oct 19 11:18:15 thinkpad dnsmasq[296]: using nameserver 75.75.75.75#53
Oct 19 11:18:15 thinkpad dnsmasq[296]: using nameserver 172.30.42.1#53
Oct 19 11:18:15 thinkpad dnsmasq[296]: ignoring nameserver 127.0.0.1 - local interface
Oct 19 11:18:15 thinkpad dnsmasq[296]: using nameserver 8.8.4.4#53
Oct 19 11:18:15 thinkpad dnsmasq[296]: using nameserver 8.8.8.8#53

But you are right that the function to restart dnsmasq should be in the /etc/hostsblock/rc.conf file. If you want it to be universal (work with any init) you could instead just use the following:

postprocess() {
          /bin/kill -HUP $(pgrep -x dnsmasq)
}

ondoho · 2013-10-22 03:17:11

i understand better now.
i removed the "no-resolv" entry from my dnsmasq.conf.
the problem is that resolv.conf gets recreated _without_ the first "nameserver 127.0.0.1" entry, although i have

- an extra last line "nohook resolv.conf" in dhcpcd.conf
- i added a file /etc/resolv.conf.head containing "127.0.0.1"

i'm reading up on things.
one workaround is to make resolv.conf read-only, using e.g. opendns servers.

WonderWoofy · 2013-10-22 05:17:48

ondoho wrote:

one workaround is to make resolv.conf read-only, using e.g. opendns servers.

Personally I don't like this option. But I think that is just my personal preference. If using static DNS servers is okay for you, think you could also simply put just "nameserver 127.0.0.1" and then set up the DNS servers in the dnsmasq.conf. You may have noticed that I actually have the google dns servers in my dnsmasq.conf as a kind of "just in case" thing.

NetworkManager seems to want to put 127.0.0.1 into the resolv.conf twice. So in that case, my internet still works fine because dnsmasq handles the dns queries (of course I was able to fix this by setting the nameservers manually and having dhcp resolve the address only). But the other three network management solutions seem to handle things just fine. I'm kind of a network manager service whore. So I switch between netctl, connman, wicd, and network-manager.

ondoho · 2013-10-22 17:16:53

i still don't see why i can't get a documented feature to work.
...
unconnected statements/answers to the above:

- if i manage to get dnsmasq to handle dns queries, doesn't that make some other installed package redundant?
...

- i'm not using networkmanager.
just netctl with a simple ppp script for mobile broadband.
i don't think there's much to improve?

---------------------------------------------------------------------
the whole thing is working now, dnsmasq uses ~150mb ram, but i don't need any complex addons for my browser(s) anymore -> faster startup & browsing.
and i like to see what's going through! i'm running dnsmasq as no-daemon, in a small window so i see everything that's whizzing through when i browse.

next i'd like to get the kwakd scenario working. i'll probably get back here with more questions.

one more: i like the postprocess () version without systemctl, but kill's output has to be redirected to nirvana, because the hostblock script interpretes it as "failed" otherwise (see /var/log/hostsblock.log).

postprocess() {
          /bin/kill -HUP $(pgrep -x dnsmasq) 1&2>/dev/zero
}

Last edited by ondoho (2013-10-22 17:22:39)

gaenserich · 2013-10-31 15:48:41

WonderWoofy wrote:

But you are right that the function to restart dnsmasq should be in the /etc/hostsblock/rc.conf file. If you want it to be universal (work with any init) you could instead just use the following:
postprocess() {
          /bin/kill -HUP $(pgrep -x dnsmasq)
}

That is so clever. You should submit that as a patch or fork the project on github and add that.

ondoho wrote:

do you think i have it set up properly now? next i'd have to get kwakd working (right now it's complaining [panic] that it can't open the default port).
or try the 0.0.0.0 variant.

It looks right. The thing with kwakd sounds like either a permission issue (where the user under which kwakd runs doesn't have the authority to bind to ports below 1024--it needs to bind to 80) or you already have something running that is bound to that port (an http server?). If you can't get kwakd to work, you can certainly still use the 127.0.0.1 variant instead of 0.0.0.0...I'm not sure if there is any performance different...I've only heard second hand that 0.0.0.0 works better without a pseudo-http-server.

ondoho wrote:

i'd like to point out a little glitch in this part of the documentation: the black block starting "postprocess()..." should be added to /etc/hostsblock/rc.conf, and NOT to /etc/dnsmasq.conf.

Fixed!

@ondoho: I'm not sure why you're having problems getting the 127.0.0.1 entry in resolv.conf. The solution resolv.conf.head should work, as long as dhcpcd is used (dhclient isn't being used instead, is it? Once you're on the network, try "pidof dhclient", and if a number comes up, dhclient is running)

ondoho wrote:

edit: i added "google.com" to the blacklist with hostsblock-urlchecker, and now the black.list looks like this:
adwords.google.comgoogle.com
www.google.com
:-(

That is odd...I blocked google.com via hostsblock-urlcheck, too, but I can't reproduce this issue.

ondoho wrote:

also i wouldn't rely solely on arch wiki for instructions on how to set up dnsmasq.

Probably good advice, but I find the instructions there adequate. If you know of another page out there, I will gladly include it or even have it replace the arch wiki page on hostblock's page.

ondoho · 2013-10-31 18:44:30

hello gaenserich, good that you're still active here.
i'm starting to rely on this script, in combo with dnsmasq & kwakd.
(the issue i had with kwakd was as simple as starting it with "sudo systemctl start kwakd".)
is there any way to verify kwakd is doing what it's supposed to? my hosts.block redirects to 127.0.0.1 now.

with resolv.conf, i'm still a bit unsure.
i'm connecting to the internet with netctl & pppd. it seems that makes things a bit different.
there's a script in /etc/ppp/ip-up.d that keeps overwriting my /etc/resolv.conf.
since there doesn't seem to be any config option for that, i modified the script to prepend /etc/resolv.conf.head to resolv.conf.
i have a really hackish feeling about this, like it's probably going to break after an update, so some more insight into this would be appreciated.
dhpcd is installed and not dhclient. but wether it's doing something, i don't know.
networkmanager i am using not.

WonderWoofy · 2013-10-31 22:45:48

If you want to check to see if kwakd is serving up... well, nothing, open your browser and just enter 127.0.0.1:80 in the address bar. If you get an immediate load of a blank page, then it is working just fine.

As far as the resolv.conf is concerned, I am not too familiar with pppd, and I hope that I never have to be. So I don't think that I am the best one to help you there.

In regard to the use of dhclient in general, to have 127.0.0.1 appended to the resolv.conf, edit or create /etc/dhclient.conf and put the following into it:

% cat /etc/resolv.conf
prepend domain-name-servers 127.0.0.1;

I have a feeling that Network Manager might ignore the dhclient.conf though, so you may have to use the features there in order to have that appended to the resolv.conf (or simply use dhcpcd). I feel as though I remember seeing something about this on the NetworkManager wiki page...

ondoho · 2013-11-02 09:18:13

WonderWoofy wrote:

If you want to check to see if kwakd is serving up... well, nothing, open your browser and just enter 127.0.0.1:80 in the address bar. If you get an immediate load of a blank page, then it is working just fine.

that worked. appreciated.

...not too familiar with pppd, and I hope that I never have to be.

why?

In regard to the use of dhclient in general... I have a feeling that Network Manager might ignore the dhclient.conf though

i'm using neither networkmanager not dhclient. should i?
right now i'm connecting like this and never had any problems with it. apart from what you usually get when using mobile broadband. but here in finland it's the default choice.

Arch Linux

#201 2013-06-01 01:56:29

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#202 2013-06-01 12:00:42

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#203 2013-06-01 13:25:43

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#204 2013-06-02 15:39:36

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#205 2013-06-03 14:34:45

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#206 2013-06-03 16:52:51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#207 2013-06-03 18:00:30

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#208 2013-06-03 20:33:15

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#209 2013-06-03 21:15:13

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#210 2013-06-03 21:33:24

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#211 2013-06-04 00:02:19

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#212 2013-06-04 01:25:50

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#213 2013-06-04 09:29:14

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#214 2013-06-04 15:27:10

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#215 2013-06-04 16:41:16

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#216 2013-06-04 18:28:54

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#217 2013-10-19 21:23:49

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#218 2013-10-19 23:56:31

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#219 2013-10-22 03:17:11

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#220 2013-10-22 05:17:48

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#221 2013-10-22 17:16:53

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#222 2013-10-31 15:48:41

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#223 2013-10-31 18:44:30

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#224 2013-10-31 22:45:48

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

#225 2013-11-02 09:18:13

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Board footer