hosts file

sadi · 2012-05-04 06:33:48

@gaenserich Does your script still work in MacOSX as well?
I'm also giving a helping hand to a MacOSX user, my daughter, but I couldn't see any "cron.xyz" folders there so I'm trying Cronnix (a crontab gui) which doesn't seem as practical as the linux way.

gaenserich · 2012-05-04 12:43:43

@sadi. Yes, with one small modification: unzip on os x has slightly different syntax...you might switch to take out a flag or two.

Instead of cron, look at the periodic folder in the /etc directory, which is essentially the same as cron. Once you install or symlink the script into one of the directories inside of "peroidic", it should work just like a cronjob.

Also, if you use a lot of blacklists, make sure to install and configure dnsmasq. The dnsresolver on os x can't handle extremely large hosts lists and will eat up all your cpu and page out profusely otherwise.

sadi · 2012-05-04 13:17:52

@gaenserich Thank you very much for os x support :-)
My daughter has had no complaints about having an 8+MB hosts file so far.

gaenserich · 2012-05-05 16:46:50

Here's an update to my hostsblock package: https://aur.archlinux.org/packages.php?ID=58976

Update 0.2:
-Added personal blacklist functionality
-Included more info in the config file
WARNING: there has been a minor change in the config file: "blacklists" is now "blocklists".

ObliviousGmn · 2012-05-09 20:03:30

I like using the Zero Host file from http://someonewhocares.org/hosts/zero/
I don't go through many sites so this Host file is right for me.

Anyway, I couldn't find any scripts the would get it for me. So I used WinterVenom's script and modified it.
Figure I drop it here in case some might like it.

#!/bin/bash
# Script by Wintervenom
# https://bbs.archlinux.org/viewtopic.php?id=106367
# SomeoneWhoCares ZeroHost

USECOLOR="yes"
. /etc/rc.d/functions # for pretty colors

if [ ! -f /etc/hosts.local ]; then
 echo "You need to create /etc/hosts.local containing the hosts you wish to keep!"
 exit 0
fi

stat_busy "SomeoneWhoCares Host List is Downloading"
wget -c -O /tmp/hosts.txt http://someonewhocares.org/hosts/zero/hosts &>/dev/null
stat_done

stat_busy "Writing Host File"
# Write this new list over the system hosts list.  MVPS' line feeds
# are \n\r; we will be getting rid of those dirty carriage returns.
# We'll first write the user's hosts head, then the black list.
cat /etc/hosts.local > /etc/hosts
cat /tmp/hosts.txt | tr -d "\r" >> /etc/hosts
rm /tmp/hosts*.txt
stat_done

Use Pixelserv to get rid of the unable to connect banners.. https://aur.archlinux.org/packages.php?ID=58975

gaenserich · 2012-05-10 12:49:01

@ObliviousGmn: hostsblock (https://aur.archlinux.org/packages.php?ID=58976) can do that too, and actually includes the 127.0.0.1 version of that particular list as well, along with a bunch of other useful lists too. You can configure hostsblock to use 0.0.0.0 instead of 127.0.0.1 if you prefer that, even if you download lists of the latter format.

Thanks for using pixelserv, too (I'm the maintainer).

Feel free to check hostsblock, if you're interested, and providing any feedback. Just from looking at Wintervenom's script, I'm planning on including all those nice color options.

ObliviousGmn · 2012-05-10 19:33:34

gaenserich wrote:

@ObliviousGmn: hostsblock (https://aur.archlinux.org/packages.php?ID=58976) can do that too, and actually includes the 127.0.0.1 version of that particular list as well, along with a bunch of other useful lists too. You can configure hostsblock to use 0.0.0.0 instead of 127.0.0.1 if you prefer that, even if you download lists of the latter format.
Thanks for using pixelserv, too (I'm the maintainer).
Feel free to check hostsblock, if you're interested, and providing any feedback. Just from looking at Wintervenom's script, I'm planning on including all those nice color options.

I'll will look check it out soon. I'm glad the script I placed helped you somehow. I actually found pixelserv on your thread I'm glad I found it.

ontobelli · 2012-07-14 16:23:38

Hi people.

Nice work @gaenserich.

Your solution looks more complex, but with nice features. I'll try it.

ontobelli · 2012-07-14 16:28:30

This is the script I currently using.

#!/bin/bash
#
# Inspired in graysky script and HostMan
# Credit to Ontobelli for this script
# Version: 2012-07-14 16:25 UTC
#

# make hosts temporal directory
HOSTSDIR=/root/.hu

# set output file
OUTPUTFILE=/etc/hosts

if [ ! -f "${HOSTSDIR}/local" ]; then
 echo "You need to create "${HOSTSDIR}/local" containing the hosts you wish to keep!"
 exit 0
fi

wget -c -O "${HOSTSDIR}/hosts.minenu" "http://hostsfile.mine.nu/Hosts"
wget -c -O "${HOSTSDIR}/hosts.hphosts" "http://hosts-file.net/download/hosts.txt"
wget -O "${HOSTSDIR}/hosts.adservers" "http://hosts-file.net/ad_servers.asp"
wget -O "${HOSTSDIR}/hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
wget -O "${HOSTSDIR}/hosts.sysctl" "http://sysctl.org/cameleon/hosts"
wget -O "${HOSTSDIR}/hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
wget -O "${HOSTSDIR}/hosts.sowc" "http://someonewhocares.org/hosts/hosts"
wget -O "${HOSTSDIR}/hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
wget -O "${HOSTSDIR}/hosts.malware" "http://www.malwaredomainlist.com/hostslist/hosts.txt"

# hosts header
echo -e "127.0.0.1 localhost ${HOSTNAME}" > "${OUTPUTFILE}"
# The following lines are desirable for IPv6 capable hosts
echo -e "::1       localhost ${HOSTNAME} ip6-localhost ip6-loopback" >> "${OUTPUTFILE}"
echo -e "fe00::0 ip6-localnet" >> "${OUTPUTFILE}"
echo -e "ff00::0 ip6-mcastprefix" >> "${OUTPUTFILE}"
echo -e "ff02::1 ip6-allnodes" >> "${OUTPUTFILE}"
echo -e "ff02::2 ip6-allrouters" >> "${OUTPUTFILE}"
echo -e "ff02::3 ip6-allhosts" >> "${OUTPUTFILE}"

# hosts body
cat "${HOSTSDIR}/local" \
    "${HOSTSDIR}/hosts.minenu" \
    "${HOSTSDIR}/hosts.hphosts" \
    "${HOSTSDIR}/hosts.adservers" \
    "${HOSTSDIR}/hosts.partial" \
    "${HOSTSDIR}/hosts.sysctl" \
    "${HOSTSDIR}/hosts.mvps" \
    "${HOSTSDIR}/hosts.sowc" \
    "${HOSTSDIR}/hosts.yoyo" \
    "${HOSTSDIR}/hosts.malware" | \
    sed -e 's/	/ /g' | \
    grep '127.0.0.1 ' | \
    tr -d "\r" | \
    tr -s [:space:] | \
    cut -d ' ' -f -2 | \
    sort | uniq | \
    grep -vf "${HOSTSDIR}/whitelist" | \
    sed -e 's/127.0.0.1 /0.0.0.0 /g' >> "${OUTPUTFILE}"

Last edited by ontobelli (2012-07-14 16:29:22)

ontobelli · 2012-07-15 03:35:31

New version:

Changed hostdir to /etc/hosts.d/
Changed wget to curl --compressed
Added support to dnsmasq

#!/bin/bash
#
# Inspired in graysky hostsupdate script, gaenserich hostsblock script and HostMan
# [c] Ontobelli 
# Version: 2012-07-15 00:00 UTC
#

# CHECK FOR NEEDED AND OPTIONAL UTILITIES AND PERMISSIONS
if [ `whoami` != "root" ]; then
    echo "Insufficient permissions. Run as root."
fi

HOSTSDIR=/etc/hosts.d/
TMPDIR=/tmp/hosts.d/
OUTPUTFILE=/etc/hosts

if [ ! -f "${HOSTSDIR}hosts.local" ]; then
 echo "You need to create "${HOSTSDIR}hosts.local" containing the hosts you wish to keep!"
 exit 0
fi

mkdir -p ${TMPDIR}

curl --compressed -o "${TMPDIR}hosts.hphosts" "http://hosts-file.net/download/hosts.txt"
curl --compressed -o "${TMPDIR}hosts.ismeh" "http://www.ismeh.com/HOSTS"
curl --compressed -o "${TMPDIR}hosts.malware" "http://www.malwaredomainlist.com/hostslist/hosts.txt"
curl --compressed -o "${TMPDIR}hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
curl --compressed -o "${TMPDIR}hosts.modyouri" "http://www.modyouri.com/adblock_hosts/hosts"
curl --compressed -o "${TMPDIR}hosts.sowc" "http://someonewhocares.org/hosts/hosts"
curl --compressed -o "${TMPDIR}hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
curl --compressed -o "${TMPDIR}hosts.sysctl" "http://sysctl.org/cameleon/hosts"
curl --compressed -o "${TMPDIR}hosts.securemecca" "http://www.securemecca.com/Downloads/hosts.txt"
curl --compressed -o "${TMPDIR}hosts.minenu" "http://hostsfile.mine.nu/Hosts"
curl --compressed -o "${TMPDIR}hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
curl --compressed -o "${TMPDIR}hosts.adservers" "http://hosts-file.net/ad_servers.asp"

cat "${TMPDIR}hosts.modyouri" | sed -e 's/0.0.0.0 /127.0.0.1 /g' > "${TMPDIR}hosts.modyouri2"

# hosts header
echo -e "127.0.0.1	${HOSTNAME} localhost" > "${OUTPUTFILE}"
echo -e "::1		${HOSTNAME} localhost ip6-localhost ip6-loopback" >> "${OUTPUTFILE}"
cat "${HOSTSDIR}hosts.local"  >> "${OUTPUTFILE}"

cat "${TMPDIR}hosts.hphosts" \
    "${TMPDIR}hosts.adservers" \
    "${TMPDIR}hosts.partial" \
    "${TMPDIR}hosts.mvps" \
    "${TMPDIR}hosts.minenu" \
    "${TMPDIR}hosts.sysctl" \
    "${TMPDIR}hosts.sowc" \
    "${TMPDIR}hosts.ismeh" \
    "${TMPDIR}hosts.modyouri2" \
    "${TMPDIR}hosts.securemecca" \
    "${TMPDIR}hosts.malware" \
    "${TMPDIR}hosts.yoyo" | \
    sed -e 's/	/ /g' | \
    grep -e '127.0.0.1 ' | \
    tr -d "\r" | \
    tr -s [:space:] | \
    cut -d ' ' -f -2 | \
    sort | uniq | \
    grep -vf "${HOSTSDIR}hosts.whitelist" | \
    sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

rm -f -r ${TMPDIR}

# restart services
/etc/rc.d/dnsmasq restart

Tested the gaenserich hostsblock script and works fine. But there is some garbage

127.0.0.1
127.0.0.1 .ads.aapl.shazamid.com
127.0.0.1 .atdmt.com
127.0.0.1 .cloudfront.net
127.0.0.1 .hitbox.com
127.0.0.1 .imeem.com
127.0.0.1 .m-adx.com
127.0.0.1 .medialytics.com
127.0.0.1 .mydas.mobi

and a "risky" redirection to some sites that could be used to hijack

255.255.255.255 broadcasthost
91.121.24.50 appulous.org
91.121.24.50 apt.appulous.org
91.121.24.50 apt.crackulous.net
91.121.24.50 apt.hackulous.org
91.121.24.50 apt.haklab.com
91.121.24.50 apt.installous.com
91.121.24.50 apt.themonkeysball.com
91.121.24.50 crackulous.net
91.121.24.50 cydia.appulous.org
91.121.24.50 cydia.crackulous.net
91.121.24.50 cydia.hackulous.org
91.121.24.50 cydia.haklab.com
91.121.24.50 cydia.installous.com
91.121.24.50 cydia.themonkeysball.com
91.121.24.50 ftp.appulous.org
91.121.24.50 ftp.crackulous.net
91.121.24.50 ftp.hackulous.org
91.121.24.50 ftp.haklab.com
91.121.24.50 ftp.installous.com
91.121.24.50 ftp.themonkeysball.com
91.121.24.50 hackulous.org
91.121.24.50 haklab.com
91.121.24.50 installous.com
91.121.24.50 themonkeysball.com
91.121.24.50 www.appulous.org
91.121.24.50 www.crackulous.net
91.121.24.50 www.hackulous.org
91.121.24.50 www.haklab.com
91.121.24.50 www.installous.com
91.121.24.50 www.themonkeysball.com

sadi · 2012-07-15 18:05:06

Thank you very much.
I have 2 questions:
1) It seems the redirections present in the hostsblock file produced by gaenserich's script come from here: http://www.modyouri.com/adblock_hosts/hosts
How come they are filtered out by your script?
2) What is the benefit of using curl's "--compressed" option?

ontobelli · 2012-07-16 05:40:21

sadi wrote:

Thank you very much.
I have 2 questions:
1) It seems the redirections present in the hostsblock file produced by gaenserich's script come from here: http://www.modyouri.com/adblock_hosts/hosts
How come they are filtered out by your script?

1) I cat all sources.
2) Standardize all of them without tabs. >>> sed -e 's/ / /g'
3) Filter all the input to select ONLY lines containing "127.0.0.1 " >>> grep -e '127.0.0.1 '
In that way I avoid the inclusion of possible unwanted redirections
4) Convert all input to unix line terminators >>> tr -d "\r"
5) Eliminate all extra spaces >>> tr -s [:space:]
6) Eliminate all comments and junk after the second term >>> cut -d ' ' -f -2
7) Sort and elimination of duplicated lines >>> sort | uniq
8) Elimination of whitelisted domains and unwanted junk >>> grep -vf "${HOSTSDIR}hosts.whitelist"
9) Change the 127.0.0.1 to 0.0.0.0 and space for a tab to readability >>> sed -e 's/127.0.0.1 /0.0.0.0 /g' >> "${OUTPUTFILE}"

Order is important and I tested the commands to guarantee the quality of the final hosts file.

2) What is the benefit of using curl's "--compressed" option?

Most servers can compress the text files that send to the client.

In that way requesting a TEXT file is almost the same that if you download a ZIP file.

It's zipped for transfer and unzipped automatically and in transparent way to the user by curl internal routines.

More or less what the @gaenserich script do with zip/unzip but without the need to code anything.

sadi · 2012-07-16 06:29:47

Muchos gracias Ontobelli!
You're a very good teacher :-)

broken pipe · 2012-07-17 08:44:18

thx a lot! really nice and clean script!

is it recommend to use dnsmasq to speed things up, because of the 8,0M hosts file?

pls add a file existence check for '/etc/hosts.d/hosts.whitelist' otherwise the script just exits after downloading the hosts lists

ontobelli · 2012-07-17 12:33:50

broken pipe wrote:

is it recommend to use dnsmasq to speed things up, because of the 8,0M hosts file?

No, dnsmasq is not needed . I have tested the 8 MB hosts file with a Pentium III with 384 MB of RAM and it works fine with Windows XP and Linux.

Resolves in miliseconds. Test by yourself

But if you want to save a few miliseconds use dnsmasq.

pls add a file existence check for '/etc/hosts.d/hosts.whitelist' otherwise the script just exits after downloading the hosts lists

I have added in the script the creation of a default whitelist in case it doesn't exist.

#!/bin/bash
#
#
# Inspired in graysky hostsupdate script, gaenserich hostsblock script and HostMan
# [c] Ontobelli  Version: 2012-07-17 12:16 UTC
#

# CHECK FOR NEEDED AND OPTIONAL UTILITIES AND PERMISSIONS
if [ `whoami` != "root" ]; then
    echo "Insufficient permissions. Run as root."
fi

HOSTSDIR=/etc/hosts.d/
TMPDIR=/tmp/hosts.d/
OUTPUTFILE=/etc/hosts

if [ ! -f "${HOSTSDIR}hosts.local" ]; then
 echo "You need to create "${HOSTSDIR}hosts.local" containing the hosts you wish to keep!"
 exit 0
fi

if [ ! -f "${HOSTSDIR}hosts.whitelist" ]; then
 echo -e "#" > "${HOSTSDIR}hosts.whitelist"
 echo -e "::1" >> "${HOSTSDIR}hosts.whitelist"
 echo -e "127.0.0.1 127.0.0.1" >> "${HOSTSDIR}hosts.whitelist"
 echo -e "127.0.0.1 localhost" >> "${HOSTSDIR}hosts.whitelist"
 exit 0
fi

mkdir -p ${TMPDIR}

curl --compressed -v -o "${TMPDIR}hosts.hphosts" "http://hosts-file.net/download/hosts.txt"
curl --compressed -v -o "${TMPDIR}hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
curl --compressed -v -o "${TMPDIR}hosts.adservers" "http://hosts-file.net/ad_servers.asp"
curl --compressed -v -o "${TMPDIR}hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
curl --compressed -v -o "${TMPDIR}hosts.minenu" "http://hostsfile.mine.nu/Hosts"
curl --compressed -v -o "${TMPDIR}hosts.malware" "http://www.malwaredomainlist.com/hostslist/hosts.txt"
curl --compressed -v -o "${TMPDIR}hosts.sysctl" "http://sysctl.org/cameleon/hosts"
curl --compressed -v -o "${TMPDIR}hosts.sowc" "http://someonewhocares.org/hosts/hosts"
curl --compressed -v -o "${TMPDIR}hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
curl --compressed -v -o "${TMPDIR}hosts.ismeh" "http://www.ismeh.com/HOSTS"
curl --compressed -v -o "${TMPDIR}hosts.securemecca" "http://www.securemecca.com/Downloads/hosts.txt"
curl --compressed -v -o "${TMPDIR}hosts.modyouri" "http://www.modyouri.com/adblock_hosts/hosts"

cat "${TMPDIR}hosts.modyouri" | sed -e 's/0.0.0.0 /127.0.0.1 /g' > "${TMPDIR}hosts.modyouri2"

# hosts header
echo -e "127.0.0.1	${HOSTNAME} localhost" > "${OUTPUTFILE}"
echo -e "::1		${HOSTNAME} localhost ip6-localhost ip6-loopback" >> "${OUTPUTFILE}"
cat "${HOSTSDIR}hosts.local"  >> "${OUTPUTFILE}"

cat "${TMPDIR}hosts.hphosts" \
    "${TMPDIR}hosts.adservers" \
    "${TMPDIR}hosts.partial" \
    "${TMPDIR}hosts.mvps" \
    "${TMPDIR}hosts.minenu" \
    "${TMPDIR}hosts.sysctl" \
    "${TMPDIR}hosts.sowc" \
    "${TMPDIR}hosts.ismeh" \
    "${TMPDIR}hosts.modyouri2" \
    "${TMPDIR}hosts.securemecca" \
    "${TMPDIR}hosts.malware" \
    "${TMPDIR}hosts.yoyo" | \
    sed -e 's/	/ /g' | \
    grep -e '127.0.0.1 ' | \
    tr -d "\r" | \
    tr -s [:space:] | \
    cut -d ' ' -f -2 | \
    sort | uniq | \
    grep -vf "${HOSTSDIR}hosts.whitelist" | \
    sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

rm -f -r ${TMPDIR}

# restart services
/etc/rc.d/dnsmasq restart

This is the hosts.whitelist I current use. "#" is used to eliminate some comments.

#
::1
127.0.0.1 127.0.0.1
127.0.0.1 localhost
127.0.0.1 bit.ly
127.0.0.1 bitly.com
127.0.0.1 goo.gl
127.0.0.1 t.co
127.0.0.1 tinyurl.com
127.0.0.1 dl.dropbox.com
127.0.0.1 dropbox.com
127.0.0.1 www.dropbox.com
127.0.0.1 pastebin.com
127.0.0.1 www.pastebin.com
127.0.0.1 opensubtitles.org
127.0.0.1 www.opensubtitles.org
127.0.0.1 ompldr.org
127.0.0.1 www.ompldr.org
127.0.0.1 viddler.com
127.0.0.1 www.viddler.com
127.0.0.1 malwaredomainlists.com
127.0.0.1 www.malwaredomainlists.com
127.0.0.1 filefactory.com
127.0.0.1 www.filefactory.com
127.0.0.1 fileserve.com
127.0.0.1 www.fileserve.com
127.0.0.1 hotfile.com
127.0.0.1 www.hotfile.com
127.0.0.1 mediafire.com
127.0.0.1 www.mediafire.com
127.0.0.1 .ads.aapl.shazamid.com
127.0.0.1 .atdmt.com
127.0.0.1 .cloudfront.net
127.0.0.1 .hitbox.com
127.0.0.1 .imeem.com
127.0.0.1 .m-adx.com
127.0.0.1 .medialytics.com
127.0.0.1 .mydas.mobi
.bp.blogspot.com

Cheers.

ontobelli · 2012-08-04 07:21:00

New version.

Change /tmp to /dev/shm/ could make a difference in other distros.
Removed MODYOURI list because bad quality.

#!/bin/bash
#
#
# Inspired in graysky hostsupdate script, gaenserich hostsblock script and HostMan
# [c] Ontobelli  Version: 2012-08-04 06:41 UTC
#

# CHECK FOR NEEDED AND OPTIONAL UTILITIES AND PERMISSIONS
if [ `whoami` != "root" ]; then
    echo "Insufficient permissions. Run as root."
fi

HOSTSDIR=/etc/hosts.d/
TMPDIR=/dev/shm/hosts.d/
OUTPUTFILE=/etc/hosts

if [ ! -f "${HOSTSDIR}hosts.local" ]; then
 echo "You need to create "${HOSTSDIR}hosts.local" containing the hosts you wish to keep!"
 exit 0
fi

if [ ! -f "${HOSTSDIR}hosts.whitelist" ]; then
 echo -e "#" > "${HOSTSDIR}hosts.whitelist"
 exit 0
fi

mkdir -p ${TMPDIR}

echo Downloading hphosts
curl --compressed -o "${TMPDIR}hosts.hphosts" "http://hosts-file.net/download/hosts.txt"
echo Downloading hphosts adservers
curl --compressed -o "${TMPDIR}hosts.adservers" "http://hosts-file.net/ad_servers.asp"
echo Downloading hphosts partial
curl --compressed -o "${TMPDIR}hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
echo Downloading hosts.mvps
curl --compressed -o "${TMPDIR}hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
echo Downloading hosts.minenu
curl --compressed -o "${TMPDIR}hosts.minenu" "http://hostsfile.mine.nu/Hosts"
echo Downloading hosts.malware
curl --compressed -o "${TMPDIR}hosts.malware" "http://www.malwaredomainlist.com/hostslist/hosts.txt"
echo Downloading hosts.sysctl
curl --compressed -o "${TMPDIR}hosts.sysctl" "http://sysctl.org/cameleon/hosts"
echo Downloading hosts.sowc
curl --compressed -o "${TMPDIR}hosts.sowc" "http://someonewhocares.org/hosts/hosts"
echo Downloading hosts.ismeh
curl --compressed -o "${TMPDIR}hosts.ismeh" "http://www.ismeh.com/HOSTS"
echo Downloading hosts.securemecca
curl --compressed -o "${TMPDIR}hosts.securemecca" "http://www.securemecca.com/Downloads/hosts.txt"
echo Downloading hosts.yoyo
curl --compressed -o "${TMPDIR}hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"

# hosts header
echo -e "127.0.0.1	${HOSTNAME} localhost.localdomain localhost loopback" > "${OUTPUTFILE}"
echo -e "::1		${HOSTNAME} localhost.localdomain localhost loopback ip6-localhost ip6-loopback" >> "${OUTPUTFILE}"
cat "${HOSTSDIR}hosts.local"  >> "${OUTPUTFILE}"

cat "${TMPDIR}hosts.hphosts" \
    "${TMPDIR}hosts.adservers" \
    "${TMPDIR}hosts.partial" \
    "${TMPDIR}hosts.mvps" \
    "${TMPDIR}hosts.minenu" \
    "${TMPDIR}hosts.malware" \
    "${TMPDIR}hosts.sysctl" \
    "${TMPDIR}hosts.sowc" \
    "${TMPDIR}hosts.ismeh" \
    "${TMPDIR}hosts.securemecca" \
    "${TMPDIR}hosts.yoyo" | \
sed -e 's/	/ /g' | \
grep -e '127.0.0.1 ' | \
tr -d "\r" | \
tr -s [:space:] | \
cut -d " " -f -2 | \
sort | uniq | \
grep -vf "${HOSTSDIR}hosts.whitelist" | \
sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

rm -f -r ${TMPDIR}

/etc/rc.d/dnsmasq restart

/etc/hosts.d/hosts.whitelist

#
127.0.0.1 bit.ly
127.0.0.1 is.gd
127.0.0.1 goo.gl
127.0.0.1 t.co
127.0.0.1 tinyurl.com
127.0.0.1 hosts-file.net
127.0.0.1 www.hosts-file.net
127.0.0.1 malwaredomainlists.com
127.0.0.1 www.malwaredomainlists.com
127.0.0.1 dl.dropbox.com
127.0.0.1 dropbox.com
127.0.0.1 www.dropbox.com
127.0.0.1 opensubtitles.org
127.0.0.1 www.opensubtitles.org
127.0.0.1 ompldr.org
127.0.0.1 www.ompldr.org
127.0.0.1 viddler.com

ontobelli · 2012-08-06 07:38:36

New version.

+ Creation of default white list and local list.
* I decided to leave a tmp folder cache because sometimes servers are not up and hosts file is incomplet otherwise and could be a risk.
= Some cleaning and better console messages.

#!/bin/bash
#
# Inspired in graysky hostsupdate script, gaenserich hostsblock script and HostMan
# [c] Ontobelli  Version: 2012-08-06 07:26 UTC
#

# CHECK FOR NEEDED AND OPTIONAL UTILITIES AND PERMISSIONS
if [ `whoami` != "root" ]; then
 echo "Insufficient permissions. Run as root."
 exit 0
fi

HOSTSDIR="/etc/hosts.d/"
# I decided to leave a tmp folder cache because sometimes servers are not up and
# hosts file is incomplet otherwise and could be a risk.
TMPDIR="${HOSTSDIR}tmp/"
OUTPUTFILE="/etc/hosts"

if [ ! -d "${TMPDIR}" ]; then
 mkdir -p ${TMPDIR}
fi

if [ ! -f "${HOSTSDIR}hosts.local" ]; then
 echo "You need to create "${HOSTSDIR}hosts.local" containing the hosts you wish to keep!"
 echo -e "fe00::0		ip6-localnet" > "${HOSTSDIR}hosts.local"
 echo -e "ff00::0		ip6-mcastprefix" >> "${HOSTSDIR}hosts.local"
 echo -e "ff02::1		ip6-allnodes" >> "${HOSTSDIR}hosts.local"
 echo -e "ff02::2		ip6-allrouters" >> "${HOSTSDIR}hosts.local"
 echo -e "ff02::3		ip6-allhosts" >> "${HOSTSDIR}hosts.local"
fi

if [ ! -f "${HOSTSDIR}hosts.whitelist" ]; then
 echo "You need to create "${HOSTSDIR}hosts.whitelist" containing the hosts you want to discard from the lists!"
 echo -e "#" > "${HOSTSDIR}hosts.whitelist"
fi

echo Downloading hosts.hphosts...
curl --compressed -s -o "${TMPDIR}hosts.hphosts" "http://hosts-file.net/download/hosts.txt"
echo Downloading hosts.adservers...
curl --compressed -s -o "${TMPDIR}hosts.adservers" "http://hosts-file.net/ad_servers.asp"
echo Downloading hosts.partial...
curl --compressed -s -o "${TMPDIR}hosts.partial" "http://hosts-file.net/hphosts-partial.asp"
echo Downloading hosts.mvps...
curl --compressed -s -o "${TMPDIR}hosts.mvps" "http://winhelp2002.mvps.org/hosts.txt"
echo Downloading hosts.minenu...
curl --compressed -s -o "${TMPDIR}hosts.minenu" "http://hostsfile.mine.nu/Hosts"
echo Downloading hosts.malware...
curl --compressed -s -o "${TMPDIR}hosts.malware" "http://www.malwaredomainlist.com/hostslist/hosts.txt"
echo Downloading hosts.sysctl...
curl --compressed -s -o "${TMPDIR}hosts.sysctl" "http://sysctl.org/cameleon/hosts"
echo Downloading hosts.sowc...
curl --compressed -s -o "${TMPDIR}hosts.sowc" "http://someonewhocares.org/hosts/hosts"
echo Downloading hosts.ismeh...
curl --compressed -s -o "${TMPDIR}hosts.ismeh" "http://www.ismeh.com/HOSTS"
echo Downloading hosts.securemecca...
curl --compressed -s -o "${TMPDIR}hosts.securemecca" "http://www.securemecca.com/Downloads/hosts.txt"
echo Downloading hosts.yoyo...
curl --compressed -s -o "${TMPDIR}hosts.yoyo" "http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"

# hosts header
echo -e "127.0.0.1	${HOSTNAME} localhost.localdomain localhost loopback" > "${OUTPUTFILE}"
echo -e "::1		${HOSTNAME} localhost.localdomain localhost loopback ip6-localhost ip6-loopback" >> "${OUTPUTFILE}"
cat "${HOSTSDIR}hosts.local"  >> "${OUTPUTFILE}"

cat "${TMPDIR}hosts.hphosts" \
    "${TMPDIR}hosts.adservers" \
    "${TMPDIR}hosts.partial" \
    "${TMPDIR}hosts.mvps" \
    "${TMPDIR}hosts.minenu" \
    "${TMPDIR}hosts.malware" \
    "${TMPDIR}hosts.sysctl" \
    "${TMPDIR}hosts.sowc" \
    "${TMPDIR}hosts.ismeh" \
    "${TMPDIR}hosts.securemecca" \
    "${TMPDIR}hosts.yoyo" | \
sed -e 's/	/ /g' | \
grep -e '127.0.0.1 ' | \
tr -d "\r" | \
tr -s [:space:] | \
cut -d " " -f -2 | \
sort | uniq | \
grep -vf "${HOSTSDIR}hosts.whitelist" | \
sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"  # in case you use a http server
#sed -e 's/127.0.0.1 /0.0.0.0	/g' >> "${OUTPUTFILE}"  # in case you don't use a http server

#rm -f -r ${TMPDIR}

/etc/rc.d/dnsmasq restart # in case you use dnsmasq
echo -e "Done!"

sadi · 2012-08-06 11:23:12

If I'm not mistaken it seems you have removed 2 lines from the previous processing commands as shown below; as a curious learner I wonder why?

ontobelli wrote:

#DELETED?: sed -e 's/0.0.0.0 /127.0.0.1 /g' | \
  sed -e 's/	/ /g' | \
  grep -e '127.0.0.1 ' | \
  tr -d "\r" | \
  tr -s [:space:] | \
  cut -d " " -f -2 | \
  sort | uniq | \
  grep -vf "${HOSTSDIR}hosts.whitelist" | \
#DELETED?: sed -e '/^#/d' | \
  sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

ontobelli · 2012-08-06 12:20:30

sadi wrote:

If I'm not mistaken it seems you have removed 2 lines from the previous processing commands as shown below; as a curious learner I wonder why?
ontobelli wrote:
#DELETED?: sed -e 's/0.0.0.0 /127.0.0.1 /g' | \
  sed -e 's/	/ /g' | \
  grep -e '127.0.0.1 ' | \
  tr -d "\r" | \
  tr -s [:space:] | \
  cut -d " " -f -2 | \
  sort | uniq | \
  grep -vf "${HOSTSDIR}hosts.whitelist" | \
#DELETED?: sed -e '/^#/d' | \
  sed -e 's/127.0.0.1 /127.0.0.1	/g' >> "${OUTPUTFILE}"

1) I eliminate this line "sed -e 's/0.0.0.0 /127.0.0.1 /g' | \" because it was ONLY needed by MODYOURI list which was also eliminated.

2) I decided not to change a space to a tab to eliminate one command at the end.
As i supposed most people are using dnsmasq no need to change the 127.0.0.1 to a 0.0.0.0 also.

3) I also change "sort | uniq" to "sort -u" same thing but in one command less.

Less commands lead to less processing time, and less CPU usage.

This is the current code

sed -e 's/	/ /g' | \
grep -e '127.0.0.1 ' | \
tr -d "\r" | \
tr -s [:space:] | \
cut -d " " -f -2 | \
sort -u | \
#sed -e 's/127.0.0.1 /0.0.0.0 /g' | \  # in case you don't use a http server
grep -vf "${HOSTSDIR}hosts.whitelist" >> "${OUTPUTFILE}"

sadi · 2012-08-06 12:32:52

Thanks a lot!
And changing "sort | uniq" to "sort -u" is also something new I've learned just now :-)
What about the command sed -e '/^#/d' (removing all comment lines from the files downloaded), isn't it needed anymore?

ontobelli · 2012-08-06 12:34:59

I don't see the need to have 2 "duplicated" scripts for hosts management.

The hostsblock script by gaenserich in AUR is a better choice for power users.

So the main goal for my script since now will be simplicity.

In that way users can decide which one they preferred.

ontobelli · 2012-08-06 12:39:06

sadi wrote:

What about the command sed -e '/^#/d' (removing all comment lines from the files downloaded), isn't it needed anymore?

Never was needed in my script.

I eliminate comments by adding "#" to the hosts.whitelist and the command
grep -vf "${HOSTSDIR}hosts.whitelist"

Arch Linux

#26 2012-05-04 06:33:48

Re: hosts file

#27 2012-05-04 12:43:43

Re: hosts file

#28 2012-05-04 13:17:52

Re: hosts file

#29 2012-05-05 16:46:50

Re: hosts file

#30 2012-05-09 20:03:30

Re: hosts file

#31 2012-05-10 12:49:01

Re: hosts file

#32 2012-05-10 19:33:34

Re: hosts file

#33 2012-07-14 16:23:38

Re: hosts file

#34 2012-07-14 16:28:30

Re: hosts file

#35 2012-07-15 03:35:31

Re: hosts file

#36 2012-07-15 18:05:06

Re: hosts file

#37 2012-07-16 05:40:21

Re: hosts file

#38 2012-07-16 06:29:47

Re: hosts file

#39 2012-07-17 08:44:18

Re: hosts file

#40 2012-07-17 12:33:50

Re: hosts file

#41 2012-08-04 07:21:00

Re: hosts file

#42 2012-08-06 07:38:36

Re: hosts file

#43 2012-08-06 11:23:12

Re: hosts file

#44 2012-08-06 12:20:30

Re: hosts file

#45 2012-08-06 12:32:52

Re: hosts file

#46 2012-08-06 12:34:59

Re: hosts file

#47 2012-08-06 12:39:06

Re: hosts file

Board footer