You are not logged in.

#176 2012-10-29 02:11:28

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

impact wrote:

I have this thing up and running and it works fine for the most part, except for Spotify. I am literally being spammed with visual ads and banners while using the Spotify client, including those full-screen banners that prevent you from doing anything on your computer.

Just add your personal list of blocked hosts to /etc/hostsblock/black.list

But take note that if the ads are in the same servers you need to access then you need also AdBlock or something alike to block selective content in that site.

Offline

#177 2012-10-29 06:31:39

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

impact wrote:

I have this thing up and running and it works fine for the most part, except for Spotify. I am literally being spammed with visual ads and banners while using the Spotify client, including those full-screen banners that prevent you from doing anything on your computer.

If it make a difference, I am in France so all those banners are probably on french hosts.

I think for this you need to know the URL your Spotify client is connected to. Then you can use "hostsblock-urlcheck.sh" to scan if you need to block any other (sub-)domains. As a result you might have an option to block certain sub-domains and/or external URLs that send those ads.

Offline

#178 2012-10-29 09:39:28

impact
Member
Registered: 2012-10-28
Posts: 55

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Is there a way to determine where does it connect to?

Offline

#179 2012-10-31 10:12:06

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

impact wrote:

Is there a way to determine where does it connect to?

You can perhaps get this info from the developers of the Spotify client you're using.

I wish I had a network monitoring utility that shows which programs on my computer are connecting to which internet addresses, maybe there is something like that.

Perhaps these 2 utilities might be helpful:
http://jnettop.kubs.info/wiki
http://netactview.sourceforge.net

Last edited by sadi (2012-10-31 10:37:07)

Offline

#180 2012-11-28 12:52:18

1126
Member
From: Cologne, Germany.
Registered: 2011-10-25
Posts: 12

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

This works great. I work adless with uzbl now. Thanks smile

There's a but, though: Since I work with Ruby, I make frequent use of

bundle install

, which fails. The connection is refused. Manually install gems via

gem install

works, though. When I replace the generated

/etc/hosts

with the old one,

bundle install

works as it should again.

Any idea what's wrong?

Offline

#181 2012-12-09 18:46:24

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Sorry for the long absence. A tiny change:

Version 0.11.2 (09.12.2012)
*swapped out use of md5sums for ls mod times to improve performance

Ideas for further development:
*Make use of GNU Parallel to speed up download times and processing, possibly at the risk of a perl dependency.
*Autodetect archiving utils in order to compress archived hosts file.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#182 2012-12-09 19:02:37

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@Lockheed: To make dhclient automatically add 127.0.0.1 to resolv.conf, just add the following to /etc/dhclient.conf.

prepend domain-name-servers 127.0.0.1;

complete with the trailing ";"

@everybody: Something else we all might want to consider: adsuck: https://aur.archlinux.org/packages/adsuck/, which is an alternative to dnsmasq designed specifically for what we do with hostsblock. I haven't had a chance to figure out how to make it work with dhcpcd, since it's set up is definitely more complicated than dnsmasq. I'll probably try it again soon, but it will take a bit of duck-duck-going around to figure it out.

@impact: You may want to try tcpdump, a utility that scans in and outgoing packets. I haven't used it extensively, but by just running it as root, I instantly got a wealth of info on my connections. Perhaps running

$ tcpdump | grep "Spotify"

could lend you some insight. Also try grepping your resultant hosts.block file to see if any spotify domains come up, e.g.

grep "spotify" /etc/hosts.block

You can unblock/block them via hostsblock-urlcheck and by trial-and-error figure out which is the responsible listing.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#183 2013-02-03 11:52:15

siot
Member
Registered: 2009-11-10
Posts: 27

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

It's very interesting!

I disgree about adsuck because then you can't use the persistent caching from pdnsd. Or config a dns server over an dns server...

I like hostsblock idea over a source file on pdnsd

Offline

#184 2013-02-07 21:14:31

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

True, and I'm leary of switching over to adsuck since it is just in the aur, not in a normal, vetted repository...I try to minimize my use of the AUR as much as possible. Ideally, though, it would be nice if hostsblock could provide multiple formats of files, i.e. not just HOST-style, but iptables or even browser adblock lists as http://pgl.yoyo.org/adservers/ already does. Alas...I won't have much time to fiddle with this in the near future though...I will gladly take patches if anyone is interested or even hand over maintainership if someone is interested in really talking care of this.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#185 2013-02-16 13:11:09

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

After some time I've begun tinkering with this wonderful script again.

1) As I prefer looking back a little further than just yesterday I decided to add log info to the previous instead of replacing it by adding one more ">" sign to this line:
exec >> "$logfile" 2>&1
But it will probably be better if new log details are added to the top, and also, if it's limited somehow, e.g. number of lines, file size, etc.

2) At some stage the script stopped displaying info in terminal; I wonder where I should look to make it display those messages again.

Offline

#186 2013-02-26 18:09:29

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@sadi: Thanks for the complements and the work you've been doing on your own!

Re: 1) Restricting the size of the logfile is typically handled by something like logrotate or systemd (if you start it via systemd and disable logfile output).
Re: 2) As long as you have the logfile option enabled, there shouldn't be any info printing to the terminal...all stdout goes to the logfile.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#187 2013-03-01 05:16:27

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

I just started using your script here, and I have to say I am pretty impressed.  For a while I was using a simply /etc/hosts file (thanks graysky), and with pdnsd it was pretty fast.  But I think that your solution of combining several block lists together seems much more complete.  I also switched to dnsmasq since it really is the loading of the blocklist that I was after, so persistent caching wasn't entirely necessary.

So I figure writing to this thread will simply garner just that much more attention to this awesomeness.

Offline

#188 2013-03-02 13:50:43

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@WonderWoofy: Thanks for the compliments as well! Stay tuned...if I ever get more time, I would like to re-impliment hostsblock in perl or python, which should hopefully drastically improve performance.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#189 2013-04-08 18:44:30

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

They are appearing some non-ascii chars in the /etc/hosts

á é í ó ú à è ì ò ù â ê î ô û

Not sure if this is caused by mixed ISO8859-x and UTF-8 input files.

And I think is better to invert the order for blacklist & whitelist to assure that what you blacklisted remains blacklisted at the end, now is overridden by whitelist.

I need to whitelist some strings but maintain blacklisted a sub-string.

Ex.

whitelist > domain.net

blacklist > sub.domain.net

but now the whitelist "domain.net" delete the line "sub.domain.net" because is processed later.

I hope I have been clear. smile

Offline

#190 2013-05-31 14:30:23

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@gaenserich, the /usr -> /usr/bin move is near!  Could you please update your pkgbuild to move the scripts out of /usr/sbin so that it doesn't conflict with the coming change?

Offline

#191 2013-05-31 20:13:16

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@WonderWoofy: Thanks for the reminder...things have been hectic here with a new baby. The next version bump will include the changes.

@ontobelli: I've made a quick fix. Give it a try to see if it works.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#192 2013-05-31 20:24:05

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.12 (31.05.2013)
*changed order of processing of black and while lists.

Version 0.11.3 (31.05.2013)
*changed installation path according to new Arch Linux practices.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#193 2013-05-31 21:15:58

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Yeah, by "near" I meant "today for the testing repo"... Really it was very easy to change it... I wanted to do the update so I just rebuilt it simply by replacing "sbin" with "bin" and all is well.

Offline

#194 2013-05-31 22:59:10

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

@ontobelli: I've made a quick fix. Give it a try to see if it works.

Congratulations for the new baby!

Thanks for the update.

Some bugs in the whitelist processing

    Local whitelist...preparedsed: -e expression #1, char 24: unknown option to `s'
    Local blacklist...prepared
DONE.

Offline

#195 2013-05-31 23:02:02

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@ontobelli, ir would probably help if you showed what your whitelist included. 

@gaenserich, I forgot to also congradulate you on the new baby in my last post.  Congrats!

Offline

#196 2013-05-31 23:11:24

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

This is what I did and is working without errors

I changed the order of     

# INCLUDE LOCAL BLACKLIST FILE

before

# GENERATE WHITELIST SED SCRIPT

    # DETERMINE THE REDIRECT URL NOT BEING USED
    if [ "$redirecturl" == "127.0.0.1" ]; then
        notredirect="0.0.0.0"
    else
        notredirect="127.0.0.1"
    fi

    # PROCESS BLOCKLIST ENTRIES INTO TARGET FILE
    if [ "$hostshead" == "0" ]; then
        rm "$hostsfile"
    else
        cp -f "$hostshead" "$hostsfile"
    fi

    # INCLUDE LOCAL BLACKLIST FILE
    printf "\n    Local blacklist..."
    cat "$blacklist" |\
    sed "s|^|$redirecturl |g" >> "$tmpdir"/hostsblock/hosts.block.d/hosts.block.0 && printf "prepared" || printf "FAILED"

    # GENERATE WHITELIST SED SCRIPT
    printf "\n    Local whitelist..."
    cat "$whitelist" |\
    sed -e 's/.*/\/&\/d/' -e 's/\./\\./g' >> "$tmpdir"/hostsblock/whitelist.sed && printf "prepared" || printf "FAILED"

    printf "\nDONE.\n\nProcessing files..."
    # DETERMINE WHETHER TO INCLUDE REDIRECTIONS
    if [ "$redirects" == "1" ]; then
        grep_eval='grep -Ih -- "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" "$tmpdir"/hostsblock/hosts.block.d/*'
    else
        grep_eval='grep -IhE -- "^127\.0\.0\.1|^0\.0\.0\.0" "$tmpdir"/hostsblock/hosts.block.d/*'
    fi

    # PROCESS AND WRITE TO FILE
    eval $grep_eval | sed -e 's/[[:space:]][[:space:]]*/ /g' -e "s/\#.*//g" -e "s/[[:space:]]$//g" -e \
    "s/$notredirect/$redirecturl/g" | sort -u | sed -f "$tmpdir"/hostsblock/whitelist.sed >> "$hostsfile" && printf "done\n"

Offline

#197 2013-05-31 23:13:55

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,412

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

The thing is, mine is working without errors as is.  So I think it may have someting to do with your whitelist file.  But unless you post it, there will be no way of telling.

Offline

#198 2013-05-31 23:15:16

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

WonderWoofy wrote:

@ontobelli, ir would probably help if you showed what your whitelist included.

white.list
That's the way I tried to eliminate non-ascii chars from the output. And it works smile

á
é
í
ó
ú
0xC30xA1
0xC30xA9
0xC30xAD
0xC30xB3
0xC30xBA
à
è
ì
ò
ù
â
ê
î
ô
û
0xC30xA2
0xC30xAA
0xC30xAE
0xC30xB4
0xC30xBB
 cdn.banners.scubl.com
 Botnet
 D
 FastFlux
 PNAP-WDC002
 hosts-file.net
 hostsfile.mine.nu
 hostsfile.org
 pgl.yoyo.org
 rlwpx.free.fr
 someonewhocares.org
 sysctl.org
 winhelp2002.mvps.org
 malwaredomainlist.com
 bit.ly
 www.bit.ly
 bitly.com
 www.bitly.com
 ow.ly
 www.ow.ly
 is.gd
 www.is.gd
 goo.gl
 www.goo.gl
 t.co
 www.t.co
 tinyurl.com
 www.tinyurl.com
 ge.tt
 www.ge.tt
 identi.ca
 www.identi.ca
 1.bp.blogspot.com
 2.bp.blogspot.com
 3.bp.blogspot.com
 4.bp.blogspot.com
 google.com.mx
 www.google.com.mx
 clients.google.com
 clients1.google.com
 clients2.google.com
 clients3.google.com
 clients4.google.com
 clients5.google.com
 blogger.com
 www.blogger.com
 youtu.be
 www.youtube.com
 www.youtube-nocookie.com
 platform.twitter.com
 blogs.msdn.com
 maximumpc.com
 www.maximumpc.com
 download.softpedia.com
 download.softpedia.ro
 download1us.softpedia.com
 games.softpedia.com
 mac.softpedia.com
 news.softpedia.com
 rarlab.com
 www.rarlab.com
 ostatic.com
 www.ostatic.com
 ycombinator.com
 news.ycombinator.com
 www.ycombinator.com
 lwn.net
 www.lwn.net
 lkml.org
 www.lkml.org
 pcworld.com
 www.pcworld.com
 networkworld.com
 www.networkworld.com
 wiki.answers.com
 www.answers.com
 dl.dropbox.com
 dropbox.com
 www.dropbox.com
 opensubtitles.org
 www.opensubtitles.org
 ompldr.org
 www.ompldr.org
 instagr.am
 www.instagr.am
 instagram.com
 www.instagram.com
 viddler.com
 www.viddler.com
 filekicker.com
 dl.filekicker.com
 www.filekicker.com
 filefactory.com
 www.filefactory.com
 fileserve.com
 www.fileserve.com
 hotfile.com
 www.hotfile.com
 mediafire.com
 www.mediafire.com
 hidemyass.com
 4.hidemyass.com
 www.hidemyass.com
 washingtontimes.com
 www.washingtontimes.com
 i.media-imdb.com
 ia.media-imdb.com
 voxel.dl.sourceforge.net
 iweb.dl.sourceforge.net
 mediablog.sourceforge.net
 seclists.org
 pastebin.com
 www.pastebin.com
 github.com
 www.github.com
 twitter.com
 www.twitter.com
 blogs.zdnet.com
 zdnet.com
 www.zdnet.com
 zdnet.co.uk
 www.zdnet.co.uk
 zdnet.com.au
 www.zdnet.com.au
 internettrafficreport.com
 plus.google.com
 www.theregister.com
 hackaday.com
 tumblr.com
 www.tumblr.com
 pingdom.com
 royal.pingdom.com
 www.pingdom.com
 www.elnuevoherald.com
 www.miamiherald.com
 code.jquery.com
 www.itworld.com
 computerworld.com
 www.computerworld.com
 www.itwire.com
 www.engadget.com
 feedly.com
 www.feedly.com
 feedproxy.google.com
 www.cracked.com
 eluniversal.com.mx
 lists.debian.org
 static.issuu.com
 soundcloud.com
 mises.org
 phoronix.com
 www.lifehacker.com.au

Last edited by ontobelli (2013-05-31 23:15:46)

Offline

#199 2013-05-31 23:47:00

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

WonderWoofy wrote:

The thing is, mine is working without errors as is.

Try this lists and see the output.

blocklists=(
  'http://www.malwaredomainlist.com/hostslist/hosts.txt'
  'http://winhelp2002.mvps.org/hosts.zip'
  'http://someonewhocares.org/hosts/hosts'
  'http://hostsfile.org/Downloads/BadHosts.unx.zip'
  'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext'
  'http://hostsfile.mine.nu/Hosts.zip'
  'http://sysctl.org/cameleon/hosts'
  'http://support.it-mate.co.uk/downloads/hosts.zip'
  'http://www.montanamenagerie.org/hostsfile/hphosts.zip'
  'http://hosts-file.net/download/hosts.zip'
  'http://hosts-file.net/ad_servers.txt'
  'http://hosts-file.net/hphosts-partial.asp'
  'http://rlwpx.free.fr/WPFF/hrsk.7z'
  'http://rlwpx.free.fr/WPFF/htrc.7z'
  'http://rlwpx.free.fr/WPFF/hpub.7z'
)

And also in the journal

Monday 31 17:56:23 localhost1 dnsmasq[6529]: bad name at /etc/hosts line 27335

Offline

#200 2013-06-01 01:47:43

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 115
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Hi all: I just caught the error too. I just made a quick fix that should hold us over for a bit.

Version 0.12.1 (31.05.2013)
*quick fix of previous change.

Last edited by gaenserich (2013-06-01 01:49:18)


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

Board footer

Powered by FluxBB