lil' script to update adblockfilter, adblocking via /etc/hosts file

kriz · 2009-09-15 09:52:59

hi, i've recently changed to adblocking via the hosts file (which works great btw), but i was missing filtersetupdating like in firefox, so i've created with my limited scripting skills this one...

# lil' script to update /etc/hosts adblock-filter
#
#hosts adblock filter taken from this site...
wget --directory-prefix=/tmp http://www.mvps.org/winhelp2002/hosts.txt
#Backup /etc/hosts to /tmp
cp /etc/hosts /tmp
#standard static hosts file
echo '# /etc/hosts: static lookup table for host names' > /etc/hosts
echo '#' >> /etc/hosts
echo '#<ip-address>    <hostname.domain.org>    <hostname>' >> /etc/hosts
echo '127.0.0.1        localhost.localdomain    localhost' >> /etc/hosts
#add custom statc host configuration here 
echo ' ' >> /etc/hosts
echo '###Ad-Blocking###' >> /etc/hosts
cat /tmp/hosts.txt >> /etc/hosts
echo '# End of file' >> /etc/hosts
rm /tmp/hosts.txt

enjoy!

scragar · 2009-12-25 18:44:13

Sorry to bump this thread, but I was looking for a solution to something else.

I made a few modifications to this code, added some basic error checking, and made it more efficient, hope you enjoy.

#!/bin/bash

ADBLOCK='http://www.mvps.org/winhelp2002/hosts.txt';
HOSTSHEAD=/etc/hosts_head;

if [ $(whoami) != "root" ]; then
  echo "Please run as root." >&2
  exit 1
fi
if [ ! -f "${HOSTSHEAD}" ]; then
  echo "Header file for hosts missing." >&2
  exit 2
fi

if [ -z "$(curl -I "${ADBLOCK}" 2>&1 | grep '200 OK' )" ]; then
  echo "Error, return code for Adblock lists is not 200." >&2
  echo "This typically indicates an error, rather than anything else." >&2
  exit 3
fi

cp "${HOSTSHEAD}" /etc/hosts
wget -O - "${ADBLOCK}" | sed /localhost/d >> /etc/hosts

EDITED to clean up a little.

Last edited by scragar (2009-12-25 21:26:21)

Meyithi · 2009-12-25 20:37:10

Cool, have been looking into this myself but wasn't happy with the duplicate

127.0.0.1 localhost

that is always present in the downloaded file.

I've put this in at the end to remove the duplicate "127.0.0.1 localhost"

sed -i -e 's-^127.0.0.1  localhost-#-' /etc/hosts

Seems to work, is probably a more elegant way though!

scragar · 2009-12-25 20:54:03

Meyithi wrote:

Cool, have been looking into this myself but wasn't happy with the duplicate
127.0.0.1 localhost
that is always present in the downloaded file.
I've put this in at the end to remove the duplicate "127.0.0.1 localhost"
sed -i -e 's-^127.0.0.1  localhost-#-' /etc/hosts
Seems to work, is probably a more elegant way though!

Just edit the last line of my edit:

wget -O - "${ADBLOCK}" | sed /localhost/d >> /etc/hosts

Meyithi · 2009-12-25 21:07:26

Cheers!

supulton · 2010-03-20 03:29:44

Great script.

ontobelli · 2012-02-15 09:11:12

hosts_udate

#!/bin/bash
#
# 2012 Ontobelli for this script

# make hosts temporal directory
HOSTSDIR=~/.hostsupdate
mkdir -p "${HOSTSDIR}"

# make temporary directory
TMPDIR=/tmp/hostsupdate
mkdir -p "${TMPDIR}"

# set output file
OUTPUTFILE="${TMPDIR}/hosts"

# set temporal file
TMPFILE="${TMPDIR}/tmpfile"

if [ ! -f "${HOSTSDIR}/hosts.local" ]; then
 echo "You need to create "${HOSTSDIR}"/hosts.local containing the hosts you wish to keep!"
 exit 0
fi

# download the mvps.org hosts file.
wget -c -O "${HOSTSDIR}/hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
# download hpHOSTS
wget -c -O "${HOSTSDIR}/hosts.hphosts" "http://support.it-mate.co.uk/downloads/HOSTS.txt"
# download hpHOSTS Partial
wget -c -O "${HOSTSDIR}/hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
# download hpHOSTS ad/tracking servers
wget -c -O "${HOSTSDIR}/hosts.adservers" "http://hosts-file.net/ad_servers.asp"
# download the pgl.yoyo.org hosts Peter Lowe - AdServers
wget -c -O "${HOSTSDIR}/hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
# download SysCtl Cameleon hosts
wget -c -O "${HOSTSDIR}/hosts.sysctl" "http://sysctl.org/cameleon/hosts"

# cat entries in a single file
cat "${HOSTSDIR}/hosts.mvps"       > "${TMPFILE}0"
cat "${HOSTSDIR}/hosts.hphosts"   >> "${TMPFILE}0"
cat "${HOSTSDIR}/hosts.partial"   >> "${TMPFILE}0"
cat "${HOSTSDIR}/hosts.adservers" >> "${TMPFILE}0"
cat "${HOSTSDIR}/hosts.yoyo"      >> "${TMPFILE}0"
cat "${HOSTSDIR}/hosts.sysctl"    >> "${TMPFILE}0"

# tabs to space
sed -e 's/	/ /g' "${TMPFILE}0" > "${TMPFILE}1"
# find relevant lines without comments
grep ^127.0.0.1 "${TMPFILE}1" > "${TMPFILE}2"
# remove duplicate spaces
cat "${TMPFILE}2" | tr -s [:space:] > "${TMPFILE}3"
# remove carriage returns
cat "${TMPFILE}3" | tr -d "\r" > "${TMPFILE}4"
# 0.0.0.0 is nicer than constantly knocking on localhosts' door.
sed -e 's/127.0.0.1 /0.0.0.0 /g' "${TMPFILE}4" > "${TMPFILE}5"
# remove inline comments
cut -d ' ' -f -2 "${TMPFILE}5" > "${TMPFILE}6"
# sort blocklist entries and remove duplicates
sort "${TMPFILE}6" | uniq > "${TMPFILE}7"
# remove unneeded blocked sites
grep -Ev ' dl.dropbox.com| host_you_want_to_whitelist' "${TMPFILE}7" > "${TMPFILE}9"

# write the user's hosts.local to head, then the blacklists
cat "${HOSTSDIR}"/hosts.local > "${OUTPUTFILE}"
cat "${TMPFILE}9" >> "${OUTPUTFILE}"
echo -e "# end of file" >> "${OUTPUTFILE}"

# move to /etc/hosts
mv "${OUTPUTFILE}" /etc/hosts

# delete temporary directory
rm -r -f "${TMPDIR}"

hosts.local

# /etc/hosts: static lookup table for host names
#
#<ip>	<hostname.domain.org>	<hostname>
127.0.0.1 localhost.localdomain localhost YOURHOSTSNAMEHERE
::1	        localhost.localdomain localhost YOURHOSTSNAMEHERE
# YOUR PERSONAL list
# blocked list

Create an alias in your ~/.bashrc

alias hu='sudo /root/.hostsupdate/hosts_update'

Run

# hu <enter>

Script and cache must be located in /root/.hostsupdate or modify scrip accordingly

Cheers.

Last edited by ontobelli (2012-02-15 09:15:17)

el mariachi · 2012-02-15 21:11:41

this is all good, but a simple instance of urxvt takes ages to start with this method. is there a way of telling urxvt to not read the hosts file?

vwyodajl · 2012-02-15 21:43:57

ontobelli's looks similar to graysky's hosts_update in the AUR.. Very simple and easy to use and converts everything to 0.0.0.0

ewaller · 2012-02-16 04:01:02

Moderator comment:

Guys: Watch the age of the thread -- This one has been stagnant for two years. Our Policy.

Closing

Arch Linux

#1 2009-09-15 09:52:59

lil' script to update adblockfilter, adblocking via /etc/hosts file

#2 2009-12-25 18:44:13

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#3 2009-12-25 20:37:10

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#4 2009-12-25 20:54:03

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#5 2009-12-25 21:07:26

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#6 2010-03-20 03:29:44

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#7 2012-02-15 09:11:12

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#8 2012-02-15 21:11:41

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#9 2012-02-15 21:43:57

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

#10 2012-02-16 04:01:02

Re: lil' script to update adblockfilter, adblocking via /etc/hosts file

Board footer