You are not logged in.

#51 2012-06-21 14:31:29

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

re: chrome store
Go to the actual site within the chrome store from which you would install a web app and enter that url in.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#52 2012-06-21 17:03:47

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thank you so much.
I'm embarrassed that this was actually due to some DNS cache issue that I've overlooked before.
All that was needed was to remove "demandware.edgesuite.net" and "clients2.google.com" from the hosts file and restart.
I'll include these to the list of domains to be allowed together with several others: clients<number>.google.com

Offline

#53 2012-07-15 03:44:28

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Nice script gaenserich. Better than mine.  But with small issues

There is some garbage 

127.0.0.1
127.0.0.1 .ads.aapl.shazamid.com
127.0.0.1 .atdmt.com
127.0.0.1 .cloudfront.net
127.0.0.1 .hitbox.com
127.0.0.1 .imeem.com
127.0.0.1 .m-adx.com
127.0.0.1 .medialytics.com
127.0.0.1 .mydas.mobi

and a "risky" redirection to some sites that could be used to hijack

In my opinion is better to ONLY include lines with the safe  127.0.0.1 and 0.0.0.0 and not all IP's in the source hosts files.

255.255.255.255 broadcasthost
91.121.24.50 appulous.org
91.121.24.50 apt.appulous.org
91.121.24.50 apt.crackulous.net
91.121.24.50 apt.hackulous.org
91.121.24.50 apt.haklab.com
91.121.24.50 apt.installous.com
91.121.24.50 apt.themonkeysball.com
91.121.24.50 crackulous.net
91.121.24.50 cydia.appulous.org
91.121.24.50 cydia.crackulous.net
91.121.24.50 cydia.hackulous.org
91.121.24.50 cydia.haklab.com
91.121.24.50 cydia.installous.com
91.121.24.50 cydia.themonkeysball.com
91.121.24.50 ftp.appulous.org
91.121.24.50 ftp.crackulous.net
91.121.24.50 ftp.hackulous.org
91.121.24.50 ftp.haklab.com
91.121.24.50 ftp.installous.com
91.121.24.50 ftp.themonkeysball.com
91.121.24.50 hackulous.org
91.121.24.50 haklab.com
91.121.24.50 installous.com
91.121.24.50 themonkeysball.com
91.121.24.50 www.appulous.org
91.121.24.50 www.crackulous.net
91.121.24.50 www.hackulous.org
91.121.24.50 www.haklab.com
91.121.24.50 www.installous.com
91.121.24.50 www.themonkeysball.com

Thanks for share your work.

Offline

#54 2012-07-15 08:37:39

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

sadi wrote:

Re: kwakd & 0.0.0.0
Although your suggestion looked like the solution, replacing 0.0.0.0 with 127.0.0.1 unfortunately didn't make any change.
I think I'll put up with the error messages for now -- there might be problem with my manual kwakd installation as well.

Maybe is your firewall.  You need to allow inbound connections to localhost from localhost.

The reason I prefer 0.0.0.0 is that I've seen many remarks like this:
"if all your doing is blocking traffic, use 0.0.0.0 instead of calling for a loopback. 127.0.0.1 is more resource intensive than using 0.0.0.0 (which just kills the connection rather than loopin it back home). Also put the line at the end not the beginning. The first lines should always be reserved for the system entries."

I have used 0.0.0.0 for years without a problem. I think is the best solution for most people.

I think that even dnsmasq is pointless for most people.

1) DNS servers are pretty fast now  < 100 ms
2) Chrome, Firefox, Opera and probably many more caches DNS
3) The OS caches disk operations. I think there is a hi probability that the most used lines in /etc/hosts are already in memory when you need them.

In my opinion there are 2 kind of users for this script.

1) Most people, that looks for security and simplicity and for them the solution of one 1 hosts file with 0.0.0.0. entries is the best one.

2) Power users who enjoy tweaking and pushing performance to the limit.

Offline

#55 2012-07-15 17:30:13

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Thank you Ontobelli,
I think I've finally sorted out the issues with 0.0.0.0, kwakd and dnsmasq.
I now have no error messages in place of ads that are being blocked.

These two scripts have different pluses and difficult to decide which one to use ;-)
But I must admit your script produces a "cleaner" blocklist although gaenserich's script has some additional capabilities.
As a result, I've made a customized script using both, and stopped working on some GUI features (at least until some stability is reached in this script and also hoping that somebody else who is more proficient in zenity can do this)

It would perhaps be much more productive if you could make this a joint project (and possibly find a zenity guy to add a gui as well?)

Offline

#56 2012-07-16 05:50:34

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

sadi wrote:

It would perhaps be much more productive if you could make this a joint project (and possibly find a zenity guy to add a gui as well?)

I'll check this script and try to improve it.

I'll think is better to have one script that works fine, than 2 that works no so good.

Offline

#57 2012-07-22 02:57:18

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Sorry I haven't been back here in a while...I haven't been getting notification emails on this.

@sadi: I'd love to see any improvements that you may have made to my script, especially when it comes to cleaning out junk.

@ontobelli:
Re: junk. We'll have to do some debugging to see if this junk comes from upstream (the blocklists) or if it's a bug in hostsblock. I have a hard time imagining that blocklists include entries that start with a dot.

Re: redirects: I'll have to check out which default lists are redirecting. Last I noticed, only one of the provided lists (commented off in the config file) did any sort of redirection.

I think I might switch over to 0.0.0.0 then. I could then turn off kwakd and dnsmasq, two fewer things to worry about.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#58 2012-07-22 03:39:53

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Per ontobelli's points, I've removed modyouri as a default blocklist, which solved both the redirect issues and the additional junk. All is peaceful in the universe again.

Version 0.8.2 (21.07.2012) https://aur.archlinux.org/packages.php?ID=58976
*added "--compressed" option to curl commands to allow for compressed
    blocklist downloads (thanks ontobelli)
*removed the modyouri blocklist from defaults since it included
    redirection and "junk" entries (noted in rc.conf comments)
IT IS HIGHLY SUGGESTED THAT YOU REMOVE THE MODYOURI ENTRY FROM RC.CONF.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#59 2012-07-24 03:14:36

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.8.3 (23.07.2012) https://aur.archlinux.org/packages.php?ID=58976
*fixed hostsblock-urlcheck to check for both http and https domains.
*simplified hostsblock-urlcheck usage. See "hostsblock-urlcheck --help"
    for details.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#60 2012-07-24 05:13:51

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@gaenserich: I haven't done anything in addition to these 2 scripts in the sense of scripting, which is a field far beyond my present skills.
I've prefered ontobelli's script for processing the downloaded lists as it effectively filters out all the junk (including redirections from modyouri).
I very much like using 0.0.0.0 and also kwakd (to get rid of almost all error messages in place of ads blocked).

Offline

#61 2012-07-24 19:18:42

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@sadi: I don't want to mess with the contents of the blocklists other than adjusting 0.0.0.0 and/or 127.0.0.1. Redirects are useful if you live in a locality where the dns is hacked, e.g. China, and as such I'd like to leave it up to the users to trust the download lists they add.

In regards to modyouri, I think you can just leave that list out to get rid of all redirects and junk. That which remains in the list is most likely covered by the herd of other blocklists, and if you don't trust a list due to its junk and redirects, then the rest of the list may be suspect as well.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#62 2012-07-24 19:19:21

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.8.4 (24.07.2012) https://aur.archlinux.org/packages.php?ID=58976
*minor code tweeks to hostsblock-urlcheck
*colorized and more compact prompts for hostsblock-urlcheck


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#63 2012-08-04 07:06:20

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

Per ontobelli's points, I've removed modyouri as a default blocklist, which solved both the redirect issues and the additional junk. All is peaceful in the universe again.

After some debugging I get to the same conclusion and eliminate modyouri list. 

Contain too many junk:
Some hosts with port :80 unnecessary.
Some hosts starting with http://  or https://
Some hosts useless and/or dead.
Some redirections.

This are the commands  I used to clean all lists at once. They are very fast and do a very good job.
And remaining comments are cleaned adding  "#" to whitelist. Feel free to use them as you wish.

cat "${TMPDIR}hosts.hphosts" \
    "${TMPDIR}hosts.adservers" \
    "${TMPDIR}hosts.partial" \
    "${TMPDIR}hosts.mvps" \
    "${TMPDIR}hosts.minenu" \
    "${TMPDIR}hosts.malware" \
    "${TMPDIR}hosts.sysctl" \
    "${TMPDIR}hosts.sowc" \
    "${TMPDIR}hosts.ismeh" \
    "${TMPDIR}hosts.securemecca" \
    "${TMPDIR}hosts.yoyo" | \
sed -e 's/	/ /g' | \
grep -e '127.0.0.1 ' | \
tr -d "\r" | \
tr -s [:space:] | \
cut -d " " -f -2 | \
sort | uniq | \
grep -vf "${HOSTSDIR}hosts.whitelist" | \
sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

Offline

#64 2012-08-04 07:28:23

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

sadi wrote:

I very much like using 0.0.0.0 and also kwakd (to get rid of almost all error messages in place of ads blocked).

As far as I know you should use 127.0.0.1 instead of 0.0.0.0 to get to the kwakd server faster.

If a HTTP server is used 127.0.0.1 is better choice, and 0.0.0.0 if you don't have a HTTP server.

Offline

#65 2012-08-04 10:26:36

rEnr3n
Member
Registered: 2012-07-25
Posts: 39

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I can't figure out how to set this up properly. I installed hostsblock, dnsmasq, and kwakd. I didn't touch the /etc/hostsblock/rc.conf file (uses dnsmasq). I ran hostsblock and /etc/hosts.block file is populated.

I uncommented these 2 lines on /etc/dnsmasq.conf

listen-address=127.0.0.1
addn-hosts=/etc/hosts.block

Dnsmasq is running.

dnsmasq   1731  0.0  0.3  36952 24660 ?        SN   17:42   0:00 /usr/bin/dnsmasq --user=dnsmasq --pid-file=/run/dnsmasq.pid

I restarted network and dnsmasq daemons. I also tried adding custom entries in hosts.block and it seems that it doesn't work at all. My main browser is chrome. I also tested it with firefox. I need help with this.

Offline

#66 2012-08-04 14:49:11

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@rEnr3n:

Make sure /etc/resolv.conf has "nameserver 127.0.0.1" as its first line. If you are using dhcpcd (which is default for netcfg, and most other connection kits), you can have this occur automatically by creating /etc/resolv.conf.head and adding "nameserver 127.0.0.1" to it.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#67 2012-08-04 15:48:33

rEnr3n
Member
Registered: 2012-07-25
Posts: 39

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Ah, thanks.

For kwakd, the submitter said:

Note, ideally this should run as its own user/group (to restrict its privileges rather than running directly as root)

How do you do that?

Offline

#68 2012-08-04 22:21:43

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@rEnr3n: There is a How-To here, which was very helpful for me:
http://crunchbanglinux.org/forums/topic … -hostfile/

Offline

#69 2012-08-05 03:55:06

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

A new version! Like ontobelli's script, hostsblock will now clean out redirects by default (there's an option to keep them, though). Even more importantly: hostsblock will only reprocess blocklists if one or more of them has changed.

Version 0.9 (04.08.2012) https://aur.archlinux.org/packages.php?ID=58976
*cleaned up minor gliches when running under OS X
*cleaned up non-color output
*now exits if not run as root
*new behavior: only writes to target hostsfile if changes detected
*new option: allow/disallow dns redirections


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#70 2012-08-05 05:03:57

rEnr3n
Member
Registered: 2012-07-25
Posts: 39

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@sadi kwakd is already in /etc/rc.d. I could just add kwakd in daemons array. But that would make it run as root. Do you run it as root? Is it safe?

Offline

#71 2012-08-05 10:55:34

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@rEnr3n: As an inexperienced Linux user I'm not in a position to give advice on this. I just follow the reliable advice found in such forums. I don't even know if kwakd can be started as user, and not root.
It seems all kwakd does is simply returning a blank html page instead of displaying an error message like "this page cannot be accessed" when a browser cannot retrieve an advert on a website, presenting the user a cleaner web page.

Offline

#72 2012-08-05 17:35:45

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@rEnr3n: I wouldn't bother with figuring out kwakd. In the meanwhile, just mildly harrass its maintainer to get the permissions sorted out. I would imagine that kwakd can't do too much damage.

YET ANOTHER NEW VERSION. After some tinkering with curl, hostsblock now will only download and process blocklists when they require updating. This should hopefully save you all some bandwidth. I also took out the color output, since it garbles up log files, and made the stdout much cleaner and logfile friendly.

Version 0.9.1 (05.08.2012) https://aur.archlinux.org/packages.php?ID=58976
*removed color interface output options (better for logging).
*made output cleaner for logging. Enter a >>/var/log/[your logfile]
    after your crontab entry for it to enable logging.
*improved behavior: only downloads new blocklists if they have been
    changed on the server. Saves bandwidth.
*rc.conf in-text documentation updated accordingly.
*removed antequated warning from install file.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#73 2012-08-05 17:38:25

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

P.S. I would clear your cache after this update. The name format of the stored cache files has changed to be more intuitive.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#74 2012-08-05 18:39:27

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

improved behavior: only downloads new blocklists if they have been changed on the server. Saves bandwidth.

This is a very good improvement indeed.
Thanks a lot!

Offline

#75 2012-08-06 09:31:01

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Very nice changes @gaenserich.

Maybe a bug. I don't get my black.list included in the final /etc/hosts. Could someone confirm?

***

Suggest to change line 175 in hostsblock for better alignment.

from    
 printf "\n\nRunning postprocessing..."
to
 printf "\n\nRunning postprocessing...\n"

Offline

Board footer

Powered by FluxBB