You are not logged in.

#26 2012-05-13 16:38:08

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.5 (13.04.2012) https://aur.archlinux.org/packages.php?ID=58976
*Added color via /etc/rc.d/functions (see "USECOLOR" in config file to disable)
*Removed notifications from trivially short and redundant steps
*Changed default blockhosts to non-zip version of the file
*Removed pixelserv check. Use kwakd instead.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#27 2012-05-15 13:38:14

Kaliwal
Member
Registered: 2011-12-05
Posts: 24

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

Version 0.5 (13.04.2012) .

Whats with April in the date?

Offline

#28 2012-05-15 13:40:52

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

oops. Meant 13.05.2012


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#29 2012-05-20 20:20:16

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Re: cleaner output hosts file:
I mean this:
For example, firstly my updated hosts file starts with a line merely consisting of 0.0.0.0 (I prefer this to 127.0.0.0) without any domain name next to it.
Secondly, the line "127.0.0.1    localhost" gets converted to "0.0.0.0    localhost".
Thirdly, five lines marked as "desirable for IPv6 capable hosts" are wiped out.
That's why I think it might be a better idea simply to added the original hosts file at the top separately (excluded from proccessing) probably with a warning for the user in readme file that they should clean their hosts file before installing and running this script for the first time to avoid possible duplicated entries.

Re: Separate files.
I thought using 3 separate files for server lists, whitelists and blacklists could make it a little more direct and practical to make any changes in these lists from time to time instead of opening a file and then going to the relevant section, etc.
But I also have a secret objective: I think I might try and create a GUI version of your script, if you don't object, and this will make it easier ;-)

Additionally, I would also like to report that I get the following error messages when I run your script:
/usr/local/bin/hostsblock.sh: line 15: stat_busy: command not found
/usr/local/bin/hostsblock.sh: line 18: stat_done: command not found
/usr/local/bin/hostsblock.sh: line 27: stat_busy: command not found
...
/usr/local/bin/hostsblock.sh: line 62: stat_done: command not found
/usr/local/bin/hostsblock.sh: line 69: stat_busy: command not found
/usr/local/bin/hostsblock.sh: line 73: stat_done: command not found

Finally, I wonder why I see no difference although I have kwakd daemon running. Instead of white space I still get the standard "This web page is not available
Chromium's connection attempt to this.com was rejected. The website may be down or your network may not be properly configured."

Thank you very much for your help.

Offline

#30 2012-05-21 13:31:51

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

sadi wrote:

Re: cleaner output hosts file:
I mean this:
For example, firstly my updated hosts file starts with a line merely consisting of 0.0.0.0 (I prefer this to 127.0.0.0) without any domain name next to it.
Secondly, the line "127.0.0.1    localhost" gets converted to "0.0.0.0    localhost".
Thirdly, five lines marked as "desirable for IPv6 capable hosts" are wiped out.
That's why I think it might be a better idea simply to added the original hosts file at the top separately (excluded from proccessing) probably with a warning for the user in readme file that they should clean their hosts file before installing and running this script for the first time to avoid possible duplicated entries.

Good points, especially about the ipv6 stuff. I'll implement that

sadi wrote:

Re: Separate files.
I thought using 3 separate files for server lists, whitelists and blacklists could make it a little more direct and practical to make any changes in these lists from time to time instead of opening a file and then going to the relevant section, etc.

Sounds good. I'll put that in for the next version, but allow for the section as a depreciated option. That will also help with a bug with the whitelist.

sadi wrote:

But I also have a secret objective: I think I might try and create a GUI version of your script, if you don't object, and this will make it easier ;-)

Cool. Send me a patch when you get it done.

sadi wrote:

Additionally, I would also like to report that I get the following error messages when I run your script:
/usr/local/bin/hostsblock.sh: line 15: stat_busy: command not found
/usr/local/bin/hostsblock.sh: line 18: stat_done: command not found
/usr/local/bin/hostsblock.sh: line 27: stat_busy: command not found
...
/usr/local/bin/hostsblock.sh: line 62: stat_done: command not found
/usr/local/bin/hostsblock.sh: line 69: stat_busy: command not found
/usr/local/bin/hostsblock.sh: line 73: stat_done: command not found

Are you using this on something other than ArchLinux? If so, you can swap out all the special "stat" functions specific to Arch with just echos. I'll include a check in the next version.

sadi wrote:

Finally, I wonder why I see no difference although I have kwakd daemon running. Instead of white space I still get the standard "This web page is not available
Chromium's connection attempt to this.com was rejected. The website may be down or your network may not be properly configured."

That could be because you use 0.0.0.0 instead of 127.0.0.1. Try the latter to see if it works. Is there a particular reason you favor 0.0.0.0 over 127.0.0.1?


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#31 2012-05-21 13:37:02

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Revised TODO:
*preserve entries from previous or given master file
*split off whitelist and blacklist arrays into separate config files. (will resolve spaces in sed issue)
*fix errors from stat_* functions when not running on ArchLinux
*revamp unzip command to function under other *nix implementations
*add command-line tool to interactively examine a given site, provide a list of domains involved in that site (and whether they are blocked or not), and add entries to the blacklist and whitelist accordingly.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#32 2012-05-21 14:52:53

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thank you very much for your replies...

Re: stat_busy & stat_done
Yes, I've later figured out that they must be specific to ArchLinux and so I've replaced them with "echo" as an Ubuntu user.

Re: kwakd & 0.0.0.0
Although your suggestion looked like the solution, replacing 0.0.0.0 with 127.0.0.1 unfortunately didn't make any change.
I think I'll put up with the error messages for now -- there might be problem with my manual kwakd installation as well.

The reason I prefer 0.0.0.0 is that I've seen many remarks like this:
"if all your doing is blocking traffic, use 0.0.0.0 instead of calling for a loopback. 127.0.0.1 is more resource intensive than using 0.0.0.0 (which just kills the connection rather than loopin it back home). Also put the line at the end not the beginning. The first lines should always be reserved for the system entries."

Offline

#33 2012-05-23 00:01:31

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.6 (22.05.2012) (https://aur.archlinux.org/packages.php?ID=58976)
*Fixed white space issues in white list entries
*Fixed errors from stat_* functions when not running on ArchLinux
*Configuration file moved to /etc/hostsblock directory as rc.conf
*New configuration files hosts.head, black.list, and white.list
*hosts.head: contains user-defined entries that go to the top of resultant output file
*removed original file recycling. If you want to carry over entries from previous output file, include in hosts.head

TODO
*add command-line tool to interactively examine a given site, provide a list of domains involved in that site (and whether they are blocked or not), and add entries to the blacklist and whitelist accordingly.
*revamp unzip command to function under other *nix implementations


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#34 2012-05-23 00:05:04

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

sadi wrote:

Re: kwakd & 0.0.0.0
Although your suggestion looked like the solution, replacing 0.0.0.0 with 127.0.0.1 unfortunately didn't make any change.
I think I'll put up with the error messages for now -- there might be problem with my manual kwakd installation as well.

The reason I prefer 0.0.0.0 is that I've seen many remarks like this:
"if all your doing is blocking traffic, use 0.0.0.0 instead of calling for a loopback. 127.0.0.1 is more resource intensive than using 0.0.0.0 (which just kills the connection rather than loopin it back home). Also put the line at the end not the beginning. The first lines should always be reserved for the system entries."

If what you said was true about 0.0.0.0, then only 127.0.0.1 would work with kwakd and pixelserv since they serve their blank pages via loopback.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#35 2012-05-27 12:34:52

Doomcide
Member
Registered: 2011-08-22
Posts: 221

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

First of all hostsblock.sh is just great, I even wrote a little how-to on the #!-forums http://crunchbanglinux.org/forums/topic … -hostfile/. I just have one little suggestion as a debian user, please make it available somewhere else than the aur like a git-repo, that would be great smile

Offline

#36 2012-05-30 00:03:51

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@Doomcide
Re: Availability
As long as you are allowed to directly link to the tarball from the AUR, I don't see a ready need for publishing the script elsewhere.

Moreover: I noticed that you posted an alternative script in the event that the user wants another blocklist url. You can adjust this directly in rc.conf by adding or deleting entries to the "blocklists" array. No need to modify the script directly or use another script.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#37 2012-05-30 00:11:20

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@Doomcide

It works different: You can add an adress to the whitelist file in two ways. If you add 'adress.com'  it will allow just this exact adress, if you leave a space in front it will allow all adresses that contain thist string, e.g.: ' adress.com' will also allow 'adress.com/foo/bar.html'. If you added anything to the white- or blacklist you'll have to rerun hostsblock.sh and the generated hostfile just won't include the specified line. I don't notice any slow-down on the system and the author of hostsblock.sh claims it uses less resources than privoxy.

Actually this is a bit off:
'address.com' will whitelist all entries that contain 'address.com', e.g. subdomain.address.com, subdomain2.address.com, subdomain.address.com.com
' address.com' will whitelist all entries that contain ' address.com', e.g. address.com, address.com.com, BUT NOT subdomain.address.com.

Last edited by gaenserich (2012-05-30 00:12:07)


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#38 2012-05-30 17:27:59

Doomcide
Member
Registered: 2011-08-22
Posts: 221

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

@Doomcide
Re: Availability
As long as you are allowed to directly link to the tarball from the AUR, I don't see a ready need for publishing the script elsewhere.

Moreover: I noticed that you posted an alternative script in the event that the user wants another blocklist url. You can adjust this directly in rc.conf by adding or deleting entries to the "blocklists" array. No need to modify the script directly or use another script.

I know you can do that, but I wanted to show an alternative way of keeping a list up-to-date. Concerning availability I understand your point, it certainly isn't a must wink

gaenserich wrote:

Actually this is a bit off:
'address.com' will whitelist all entries that contain 'address.com', e.g. subdomain.address.com, subdomain2.address.com, subdomain.address.com.com
' address.com' will whitelist all entries that contain ' address.com', e.g. address.com, address.com.com, BUT NOT subdomain.address.com.

Thanks for clearing that up, I'll correct my post.

Offline

#39 2012-06-01 19:33:58

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

A major addition: hostsblock now also includes hostsblock-urlcheck, which will scan a given url for potential subdomains and ask the user whether these domains should be blocked or unblocked. Use it on sites that don't seem to function correctly to unblock subdomains needed for proper functionality or on sites that have unblocked ads to root out missed blacklist urls.

Version 0.7 (01.06.2012) https://aur.archlinux.org/packages.php?ID=58976
*Added hostsblock-urlcheck, which checks a given url for contained subdomains and allows the user to white or blacklist them selectively
*Fixed minor typo

TODO
*revamp unzip command to function under other *nix implementations


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#40 2012-06-02 23:22:10

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thanks for the new version.
Here are further comments, queries and suggestions:
1) If we prefer creating and using /etc/hosts.block (in addition to /etc/hosts) do we still need the host.head contents or should be inserted only when one hosts file is used?
2) Should all instances of "91.121.24.50" be replaced with "127.0.0.1" as well?
3) How about moving the URL links into a separate file (e.g. "servers.list") like the files black.list and white.list for the same purposes of editing the list more directly and easily while the other settings left in rc.conf are not likely to be changed as often as these 3 lists.

Offline

#41 2012-06-03 20:52:43

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.7.1 (03.06.2012) https://aur.archlinux.org/packages.php?ID=58976
*Made changes from hostsblock-urlcheck apply more quickly without running hostblock
*Made hosts.head file optional, and adjusted default rc.conf accordingly.

TODO
*revamp unzip command to function under other *nix implementations

@sadi:
1) Fixed this in this new version. If you put in 0 for the hostshead variable (see new default rc.conf)
2) I believe you must have subscribed to one of the redirection blocklists (used to fix dns cache poisoning https://en.wikipedia.org/wiki/DNS_spoofing). Hostblock will only change 127.0.0.1 or 0.0.0.0 entries.
3) Once you've picked your lists, the blocklists array is less likely to be modified than the white or blacklists, and it isn't too difficult to add entries to the array otherwise (with sed or awk). I'm content to keep that variable in rc.conf.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#42 2012-06-04 19:57:42

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.7.2 (04.06.2012) https://aur.archlinux.org/packages.php?ID=58976
*Fixed error in hostsblock-urlcheck that cased full-stop error

TODO
*revamp unzip command to function under other *nix implementations


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#43 2012-06-11 12:23:12

Aciid
Member
Registered: 2008-10-17
Posts: 2

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Hey, thanks for the great hostsblock concept. I really like the security and the additional adblocking capabilites it gives from the gecko. I multiple machines now going thru my DNS node because this beats OpenDNS by far.
I just noticed an additional must to add to the whitelist for blogspot if you want images to be displayed. For some reason most of blogspot's CDN's are on those lists.

"/\.bp\.blogspot\.com/d" -e \

Offline

#44 2012-06-12 17:06:42

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@Aciid. Thanks, I'll add that entry to the default whitelist.

Last edited by gaenserich (2012-06-12 17:06:56)


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#45 2012-06-14 20:55:51

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Verson 0.8 (14.06.2012) https://aur.archlinux.org/packages.php?ID=58976
*made postprocessing routine (such as dnsmasq restart) more flexible for those
    who might want to use a different program. PLEASE CHECK RC.CONF.
*minor fixes with variable quoting, console output, code formatting
*added run-time dependency checks
*updated documentation in rc.conf
*added additional entry to whitelists
*fixed error with whitelist not taking periods literally

TODO
*revamp unzip command to function under other *nix implementations


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#46 2012-06-14 21:15:48

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

[edited to reflect new behavior]
A quick guide to hostsblock-urlcheck, a command line utility which scans a given url for domains to block or unblock.

If you install hostsblock, hostsblock-urlcheck is already available. Just run it as root, e.g.:

$ hostsblock-urlcheck scan http://www.cpan.org

The utility will then scan through everything that looks like a url, check to see whether it is blocked or not, and ask you whether you would like to change that:

$ hostsblock-urlcheck scan http://www.cpan.org
NOT YET BLOCKED: 'learn.perl.org'. Block? [y/N]
NOT YET BLOCKED: 'lists.perl.org'. Block? [y/N]
NOT YET BLOCKED: 'metacpan.org'. Block? [y/N]
NOT YET BLOCKED: 'mirrors.cpan.org'. Block? [y/N]
NOT YET BLOCKED: 'perldoc.perl.org'. Block? [y/N]
BLOCKED: 'search.cpan.org'. Unblock? [y/N]y
Unblocked search.cpan.org.
NOT YET BLOCKED: 'www.cpan.org'. Block? [y/N]
NOT YET BLOCKED: 'www.perl.org'. Block? [y/N]
BLOCKED: 'www.yellowbot.com'. Unblock? [y/N]
Url check finished. Run postprocessing subroutine (e.g. dnsmasq)? [y/N]
y
:: Stopping DNS/DHCP daemon                                                                                                    [DONE] 
:: Starting DNS/DHCP daemon 

At the end here, if you have dnsmasq set up, you can have it reboot so that your changes are made. Otherwise, this utility makes changes to the system files instantaneously.

If you just want to see if a specific page is blocked by itself, just use the "page" option:

 $ hostsblock-urlcheck page http://example.com
NOT YET BLOCKED: 'example.com'. Block? [y/N]
y
Blocking example.com
Page check finished. Run postprocessing subroutine (e.g. dnsmasq)? [y/N]
y
:: Stopping DNS/DHCP daemon                                                                                                    [DONE] 
:: Starting DNS/DHCP daemon

There is still a lot of debugging to do, so give it a whirl and let me know of any problems you might encounter.

Last edited by gaenserich (2012-06-16 00:20:56)


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#47 2012-06-16 00:17:06

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.8.1 (15.06.2012) https://aur.archlinux.org/packages.php?ID=58976
*changed hostsblock-urlcheck to also check given urls as to whether they are
    blocked or not (via "page" option). This is in addition to the whole page
    scan option ("scan"), which reads through the whole given page to see if
    contained subdomains are blocked or not.
*regular behavior of hostsblock-urlcheck now relegated to "scan" option. See
    'hostsblock-urlcheck help' for details.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#48 2012-06-21 10:01:49

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

A friend of mine, who used this script upon my recommendation, complained that (1) an online shopping site (www.jackspade.com) stopped being displayed properly and (2) Google Chrome could no longer install WebApps.
Could it be because these sites are designed to work properly only when certain adservers are allowed?
Commenting out the domain names identified by hostsblock-urlcheck (demandware.edgesuite.net and www.googleadservices.com) didn't make any change.
The problem was resolved only by restoring the hosts file to its original state, i.e. no domains blocked, but this shouldn't be the ideal way of dealing with such problems.
I wonder if there is a way of knowing which particular adservers such a website requires?

Offline

#49 2012-06-21 13:59:16

gaenserich
Member
From: Pennsylvania
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@sadi
re: jackspade

root $ hostsblock-urlcheck page www.jackspade.com

will allow your friend to unblock the site.

re: chrome web store
I actually did a hostsblock-urlcheck scan on the chrome store and found the offending blocked url, although I don't have that available right now. Once I get home I'll check it out. I'll look into adding a logging utility which would help identify problematic download lists.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#50 2012-06-21 14:22:33

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thank you very much.

re: jackspade
I used hostsblock-urlcheck with "scan" option and found only "demandware.edgesuite.net" and "www.googleadservices.com" blocked, and "www.jackspade.com" not blocked. But making sure that all these three domains are not blocked didn't have any effect.

re: chrome web store
I did hostsblock-urlcheck scan for "https://chrome.google.com" with no results. I hope you have the key for this.

Offline

Board footer

Powered by FluxBB