You are not logged in.
I have this thing up and running and it works fine for the most part, except for Spotify. I am literally being spammed with visual ads and banners while using the Spotify client, including those full-screen banners that prevent you from doing anything on your computer.
Just add your personal list of blocked hosts to /etc/hostsblock/black.list
But take note that if the ads are in the same servers you need to access then you need also AdBlock or something alike to block selective content in that site.
Offline
I have this thing up and running and it works fine for the most part, except for Spotify. I am literally being spammed with visual ads and banners while using the Spotify client, including those full-screen banners that prevent you from doing anything on your computer.
If it make a difference, I am in France so all those banners are probably on french hosts.
I think for this you need to know the URL your Spotify client is connected to. Then you can use "hostsblock-urlcheck.sh" to scan if you need to block any other (sub-)domains. As a result you might have an option to block certain sub-domains and/or external URLs that send those ads.
Offline
Is there a way to determine where does it connect to?
Offline
Is there a way to determine where does it connect to?
You can perhaps get this info from the developers of the Spotify client you're using.
I wish I had a network monitoring utility that shows which programs on my computer are connecting to which internet addresses, maybe there is something like that.
Perhaps these 2 utilities might be helpful:
http://jnettop.kubs.info/wiki
http://netactview.sourceforge.net
Last edited by sadi (2012-10-31 10:37:07)
Offline
This works great. I work adless with uzbl now. Thanks
There's a but, though: Since I work with Ruby, I make frequent use of
bundle install
, which fails. The connection is refused. Manually install gems via
gem install
works, though. When I replace the generated
/etc/hosts
with the old one,
bundle install
works as it should again.
Any idea what's wrong?
Offline
Sorry for the long absence. A tiny change:
Version 0.11.2 (09.12.2012)
*swapped out use of md5sums for ls mod times to improve performance
Ideas for further development:
*Make use of GNU Parallel to speed up download times and processing, possibly at the risk of a perl dependency.
*Autodetect archiving utils in order to compress archived hosts file.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
@Lockheed: To make dhclient automatically add 127.0.0.1 to resolv.conf, just add the following to /etc/dhclient.conf.
prepend domain-name-servers 127.0.0.1;
complete with the trailing ";"
@everybody: Something else we all might want to consider: adsuck: https://aur.archlinux.org/packages/adsuck/, which is an alternative to dnsmasq designed specifically for what we do with hostsblock. I haven't had a chance to figure out how to make it work with dhcpcd, since it's set up is definitely more complicated than dnsmasq. I'll probably try it again soon, but it will take a bit of duck-duck-going around to figure it out.
@impact: You may want to try tcpdump, a utility that scans in and outgoing packets. I haven't used it extensively, but by just running it as root, I instantly got a wealth of info on my connections. Perhaps running
$ tcpdump | grep "Spotify"
could lend you some insight. Also try grepping your resultant hosts.block file to see if any spotify domains come up, e.g.
grep "spotify" /etc/hosts.block
You can unblock/block them via hostsblock-urlcheck and by trial-and-error figure out which is the responsible listing.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
It's very interesting!
I disgree about adsuck because then you can't use the persistent caching from pdnsd. Or config a dns server over an dns server...
I like hostsblock idea over a source file on pdnsd
Offline
True, and I'm leary of switching over to adsuck since it is just in the aur, not in a normal, vetted repository...I try to minimize my use of the AUR as much as possible. Ideally, though, it would be nice if hostsblock could provide multiple formats of files, i.e. not just HOST-style, but iptables or even browser adblock lists as http://pgl.yoyo.org/adservers/ already does. Alas...I won't have much time to fiddle with this in the near future though...I will gladly take patches if anyone is interested or even hand over maintainership if someone is interested in really talking care of this.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
After some time I've begun tinkering with this wonderful script again.
1) As I prefer looking back a little further than just yesterday I decided to add log info to the previous instead of replacing it by adding one more ">" sign to this line:
exec >> "$logfile" 2>&1
But it will probably be better if new log details are added to the top, and also, if it's limited somehow, e.g. number of lines, file size, etc.
2) At some stage the script stopped displaying info in terminal; I wonder where I should look to make it display those messages again.
Offline
@sadi: Thanks for the complements and the work you've been doing on your own!
Re: 1) Restricting the size of the logfile is typically handled by something like logrotate or systemd (if you start it via systemd and disable logfile output).
Re: 2) As long as you have the logfile option enabled, there shouldn't be any info printing to the terminal...all stdout goes to the logfile.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
I just started using your script here, and I have to say I am pretty impressed. For a while I was using a simply /etc/hosts file (thanks graysky), and with pdnsd it was pretty fast. But I think that your solution of combining several block lists together seems much more complete. I also switched to dnsmasq since it really is the loading of the blocklist that I was after, so persistent caching wasn't entirely necessary.
So I figure writing to this thread will simply garner just that much more attention to this awesomeness.
Offline
@WonderWoofy: Thanks for the compliments as well! Stay tuned...if I ever get more time, I would like to re-impliment hostsblock in perl or python, which should hopefully drastically improve performance.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
They are appearing some non-ascii chars in the /etc/hosts
á é í ó ú à è ì ò ù â ê î ô û
Not sure if this is caused by mixed ISO8859-x and UTF-8 input files.
And I think is better to invert the order for blacklist & whitelist to assure that what you blacklisted remains blacklisted at the end, now is overridden by whitelist.
I need to whitelist some strings but maintain blacklisted a sub-string.
Ex.
whitelist > domain.net
blacklist > sub.domain.net
but now the whitelist "domain.net" delete the line "sub.domain.net" because is processed later.
I hope I have been clear.
Offline
@gaenserich, the /usr -> /usr/bin move is near! Could you please update your pkgbuild to move the scripts out of /usr/sbin so that it doesn't conflict with the coming change?
Offline
@WonderWoofy: Thanks for the reminder...things have been hectic here with a new baby. The next version bump will include the changes.
@ontobelli: I've made a quick fix. Give it a try to see if it works.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
Version 0.12 (31.05.2013)
*changed order of processing of black and while lists.
Version 0.11.3 (31.05.2013)
*changed installation path according to new Arch Linux practices.
Check out hostsblock for system-wide ad- and malware-blocking.
Offline
Yeah, by "near" I meant "today for the testing repo"... Really it was very easy to change it... I wanted to do the update so I just rebuilt it simply by replacing "sbin" with "bin" and all is well.
Offline
@ontobelli: I've made a quick fix. Give it a try to see if it works.
Congratulations for the new baby!
Thanks for the update.
Some bugs in the whitelist processing
Local whitelist...preparedsed: -e expression #1, char 24: unknown option to `s'
Local blacklist...prepared
DONE.
Offline
@ontobelli, ir would probably help if you showed what your whitelist included.
@gaenserich, I forgot to also congradulate you on the new baby in my last post. Congrats!
Offline
This is what I did and is working without errors
I changed the order of
# INCLUDE LOCAL BLACKLIST FILE
before
# GENERATE WHITELIST SED SCRIPT
# DETERMINE THE REDIRECT URL NOT BEING USED
if [ "$redirecturl" == "127.0.0.1" ]; then
notredirect="0.0.0.0"
else
notredirect="127.0.0.1"
fi
# PROCESS BLOCKLIST ENTRIES INTO TARGET FILE
if [ "$hostshead" == "0" ]; then
rm "$hostsfile"
else
cp -f "$hostshead" "$hostsfile"
fi
# INCLUDE LOCAL BLACKLIST FILE
printf "\n Local blacklist..."
cat "$blacklist" |\
sed "s|^|$redirecturl |g" >> "$tmpdir"/hostsblock/hosts.block.d/hosts.block.0 && printf "prepared" || printf "FAILED"
# GENERATE WHITELIST SED SCRIPT
printf "\n Local whitelist..."
cat "$whitelist" |\
sed -e 's/.*/\/&\/d/' -e 's/\./\\./g' >> "$tmpdir"/hostsblock/whitelist.sed && printf "prepared" || printf "FAILED"
printf "\nDONE.\n\nProcessing files..."
# DETERMINE WHETHER TO INCLUDE REDIRECTIONS
if [ "$redirects" == "1" ]; then
grep_eval='grep -Ih -- "^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" "$tmpdir"/hostsblock/hosts.block.d/*'
else
grep_eval='grep -IhE -- "^127\.0\.0\.1|^0\.0\.0\.0" "$tmpdir"/hostsblock/hosts.block.d/*'
fi
# PROCESS AND WRITE TO FILE
eval $grep_eval | sed -e 's/[[:space:]][[:space:]]*/ /g' -e "s/\#.*//g" -e "s/[[:space:]]$//g" -e \
"s/$notredirect/$redirecturl/g" | sort -u | sed -f "$tmpdir"/hostsblock/whitelist.sed >> "$hostsfile" && printf "done\n"
Offline
The thing is, mine is working without errors as is. So I think it may have someting to do with your whitelist file. But unless you post it, there will be no way of telling.
Offline
@ontobelli, ir would probably help if you showed what your whitelist included.
white.list
That's the way I tried to eliminate non-ascii chars from the output. And it works
á
é
í
ó
ú
0xC30xA1
0xC30xA9
0xC30xAD
0xC30xB3
0xC30xBA
à
è
ì
ò
ù
â
ê
î
ô
û
0xC30xA2
0xC30xAA
0xC30xAE
0xC30xB4
0xC30xBB
cdn.banners.scubl.com
Botnet
D
FastFlux
PNAP-WDC002
hosts-file.net
hostsfile.mine.nu
hostsfile.org
pgl.yoyo.org
rlwpx.free.fr
someonewhocares.org
sysctl.org
winhelp2002.mvps.org
malwaredomainlist.com
bit.ly
www.bit.ly
bitly.com
www.bitly.com
ow.ly
www.ow.ly
is.gd
www.is.gd
goo.gl
www.goo.gl
t.co
www.t.co
tinyurl.com
www.tinyurl.com
ge.tt
www.ge.tt
identi.ca
www.identi.ca
1.bp.blogspot.com
2.bp.blogspot.com
3.bp.blogspot.com
4.bp.blogspot.com
google.com.mx
www.google.com.mx
clients.google.com
clients1.google.com
clients2.google.com
clients3.google.com
clients4.google.com
clients5.google.com
blogger.com
www.blogger.com
youtu.be
www.youtube.com
www.youtube-nocookie.com
platform.twitter.com
blogs.msdn.com
maximumpc.com
www.maximumpc.com
download.softpedia.com
download.softpedia.ro
download1us.softpedia.com
games.softpedia.com
mac.softpedia.com
news.softpedia.com
rarlab.com
www.rarlab.com
ostatic.com
www.ostatic.com
ycombinator.com
news.ycombinator.com
www.ycombinator.com
lwn.net
www.lwn.net
lkml.org
www.lkml.org
pcworld.com
www.pcworld.com
networkworld.com
www.networkworld.com
wiki.answers.com
www.answers.com
dl.dropbox.com
dropbox.com
www.dropbox.com
opensubtitles.org
www.opensubtitles.org
ompldr.org
www.ompldr.org
instagr.am
www.instagr.am
instagram.com
www.instagram.com
viddler.com
www.viddler.com
filekicker.com
dl.filekicker.com
www.filekicker.com
filefactory.com
www.filefactory.com
fileserve.com
www.fileserve.com
hotfile.com
www.hotfile.com
mediafire.com
www.mediafire.com
hidemyass.com
4.hidemyass.com
www.hidemyass.com
washingtontimes.com
www.washingtontimes.com
i.media-imdb.com
ia.media-imdb.com
voxel.dl.sourceforge.net
iweb.dl.sourceforge.net
mediablog.sourceforge.net
seclists.org
pastebin.com
www.pastebin.com
github.com
www.github.com
twitter.com
www.twitter.com
blogs.zdnet.com
zdnet.com
www.zdnet.com
zdnet.co.uk
www.zdnet.co.uk
zdnet.com.au
www.zdnet.com.au
internettrafficreport.com
plus.google.com
www.theregister.com
hackaday.com
tumblr.com
www.tumblr.com
pingdom.com
royal.pingdom.com
www.pingdom.com
www.elnuevoherald.com
www.miamiherald.com
code.jquery.com
www.itworld.com
computerworld.com
www.computerworld.com
www.itwire.com
www.engadget.com
feedly.com
www.feedly.com
feedproxy.google.com
www.cracked.com
eluniversal.com.mx
lists.debian.org
static.issuu.com
soundcloud.com
mises.org
phoronix.com
www.lifehacker.com.au
Last edited by ontobelli (2013-05-31 23:15:46)
Offline
The thing is, mine is working without errors as is.
Try this lists and see the output.
blocklists=(
'http://www.malwaredomainlist.com/hostslist/hosts.txt'
'http://winhelp2002.mvps.org/hosts.zip'
'http://someonewhocares.org/hosts/hosts'
'http://hostsfile.org/Downloads/BadHosts.unx.zip'
'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext'
'http://hostsfile.mine.nu/Hosts.zip'
'http://sysctl.org/cameleon/hosts'
'http://support.it-mate.co.uk/downloads/hosts.zip'
'http://www.montanamenagerie.org/hostsfile/hphosts.zip'
'http://hosts-file.net/download/hosts.zip'
'http://hosts-file.net/ad_servers.txt'
'http://hosts-file.net/hphosts-partial.asp'
'http://rlwpx.free.fr/WPFF/hrsk.7z'
'http://rlwpx.free.fr/WPFF/htrc.7z'
'http://rlwpx.free.fr/WPFF/hpub.7z'
)
And also in the journal
Monday 31 17:56:23 localhost1 dnsmasq[6529]: bad name at /etc/hosts line 27335
Offline
Hi all: I just caught the error too. I just made a quick fix that should hold us over for a bit.
Version 0.12.1 (31.05.2013)
*quick fix of previous change.
Last edited by gaenserich (2013-06-01 01:49:18)
Check out hostsblock for system-wide ad- and malware-blocking.
Offline