You are not logged in.

#76 2012-08-06 10:15:56

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

'http://www.securemecca.com/Downloads/hosts.txt'    # General blocklist (defunct for now)
'http://www.hostsfile.org/Downloads/hosts.txt'        # General blocklist (defunct for now)

Those are exactly the same file. Same checksum.

***

http://hosts-file.net/ad_servers.asp is not always included in the 'http://support.it-mate.co.uk/downloads/hphosts.zip' and would be a nice addition to the downloads list

Offline

#77 2012-08-06 10:16:42

itman
Member
From: Switzerland
Registered: 2010-05-21
Posts: 124

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

confirmed, nothing  but the first line got processed... in the black-list though...

and while we're at it: if all block-files are up-to-date, but you change something in the white-list, that white-list didn't get processed.

Offline

#78 2012-08-06 10:58:21

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

itman wrote:

confirmed, nothing  but the first line got processed... in the black-list though...

Thanks @itman.

@gaenserich

The line in rc.conf could be simplified just to query relevant fields.

from
'http://pgl.yoyo.org/adservers/serverlis … =plaintext'
to
'http://pgl.yoyo.org/as/serverlist.php?h … =plaintext'

Edit.

Forum shortened the URL.

Last edited by ontobelli (2012-08-06 11:00:08)

Offline

#79 2012-08-06 15:28:14

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@ontobelli: Updated the rc.conf on securemecca and commented out the redundant url. I'll swap out the pgl.yoyo.org url in the next version.
@itman: w/whitelist/blacklist changes. Oi...that's a tougher one. In the interim, delete one of the cache files to induce hostsblock to re-process. I'll fix that in the next version

I fixed up the blacklist issue, and the problem with the stdout line break was due to not having a "FAIL" option, which is now available. Test away!

Version 0.9.2 (06.08.2012) https://aur.archlinux.org/packages.php?ID=58976
*added "FAIL" notifications
*fixed issue with blacklist
*fixed issue with 7z files
*fixed documentation in rc.conf on mirrored, defunct blocklists


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#80 2012-08-06 17:41:02

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@itman: Actually, it just occurred to me. hostsblock-urlcheck takes care of this on its own. When black or whitelist something via this tool, it not only adds it to those respective files, it also removes it from the existing target hostsfile and runs your postprocess variable (e.g. dnsmasq). Using this should help, but if you prefer to do it yourself, you only have to do an

sed -i '/[whitelisted subdomain here]/d' /etc/[yourhostfile]

for a whitelist entry or

echo "[redirecturl] [blacklisted subdomain here]" >> /etc/[yourhostfile]

for a blacklist entry (plus restart dnsmasq if you have that going.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#81 2012-08-06 17:57:16

latjay
Member
Registered: 2012-06-30
Posts: 7

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Offline

#82 2012-08-06 18:21:14

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

In den englischen Foren können wir leider Einträge von anderen Sprachen gar nicht so leicht verstehen, und finden es auch ziemlich unnutzbar, wenn man eine Verknüpfung ohne weitere Erklärung hinfügt. Bitte sag's mal, was du mit der Verknüpfung meinst und sogar was die verknüpfte Webseite enthält.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#83 2012-08-06 19:33:27

latjay
Member
Registered: 2012-06-30
Posts: 7

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@gaenserich
The page I posted contains a link to another hosts file:
http://abp.mozilla-hispano.org/nauscopio/hosts.zip
You may consider adding it to the default hostsblock's list.

Offline

#84 2012-08-06 19:36:58

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@latja: Ah, I see. It looks mostly like an out-of-date copy of the someonewhocares file, but I'll include it as an option (disabled by default) in the next release.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#85 2012-08-08 11:45:58

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich

2 suggestions.

1) ONLY process the lists active (not commented) in rc.conf because actual behavior include every file in /var/cache/hostsblock even when commented in rc.conf

2) Option to have multiple hosts in a single line.

Advantages:
- Help to reduce file size considerably,
- Reduce the number of lines in the hosts file,
- Reduce memory usage,
- Improve performance a little. 

With an option in rc.conf to set the number of hosts per line.

Example with 5 hosts per line:

127.0.0.1 host1.com  host2.com host3.com host4.com host5.com
127.0.0.1 host6.com  host7.com host8.com host9.com host10.com

hpHosts call it optimized hosts file

Offline

#86 2012-08-08 22:22:05

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@ontobelli: Asked and implemented, albeit without an option to delineate the number of entries per line. Send me a patch if you can get that working.

Version 0.10 (08.08.2012) https://aur.archlinux.org/packages.php?ID=58976
*implemented an optimization option, which shrinks target hostfile
    size by putting all entries redirecting to the same address
    on the same line.
*instead of processing all files in the cache, hostsblock now only
    processes those listed (and not commented out) in rc.conf
*updated documentation in rc.conf
*added another blocklist
*simplified really long pgl.yoyo.com blocklisturl


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#87 2012-08-09 02:57:18

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

@ontobelli: Asked and implemented, albeit without an option to delineate the number of entries per line. Send me a patch if you can get that working.

Thanks. @gaenserich

1) Bug. Not processed hosts.head when optimize="1"

***

I think is a pair of for loops what you need.

First one to process all hosts input and second one inside the fist one to write each line with the number of hosts per line.

***

All hosts in a single line maybe impact performance and make very difficult to edit.

I can't even add a line to it in Kate editor.

Last edited by ontobelli (2012-08-09 03:17:22)

Offline

#88 2012-08-09 13:30:03

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@ontobelli.
Re: Bug. I know that the head file is in there somewhere, but the optimization pass messes it up or hides it somehow.

Re: for loops. An earlier implementation of the optimization pass used for-loops, but it took over a half hour for it to process the entire original file. I'm looking into using cut to insert breaks, but squeezing the $redirecturl in in front of each line becomes problematic.

Re: difficulties of the radically modified file: I have noticed a performance hit when the file is updated (dnsmasq goes crazy for a while). Perhaps there is a trade off between file size and cpu usage, and that "optimization" may be a misnomer.

Quite frankly, I consider the optimized file to be binary-file-like, i.e. uneditable directly. Instead, use system tools, e.g. to add a line:

echo "[line content]" >> /etc/[your hosts file]

Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#89 2012-08-09 22:56:55

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Fixed up the bug. The head file will still get buried when optimization is engaged, but it's there.

Version 0.10.1 (09.08.2012)
*fixed issues with optimization pass, integrated it into main
    processing pass
*fixed issue with 32-bit machines and touch

I still haven't figured out how to make arbitrary-length lines. I was going to use fold, but if -w isn't set long enough on it, it will cut-off entries. Does anyone know of a coreutils utility that will put in a line break only AFTER a set column width (with fold, it goes before). E.g.:

Fold does the following:

$ echo "www.example.com www.example.org" | fold -s -w8
www.exam
ple.com 
www.exam
ple.org

desired behavior would be:

$ echo "www.example.com www.example.org" | fold -s -w8
www.example.com
www.example.org

Last edited by gaenserich (2012-08-09 22:57:13)


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#90 2012-08-09 23:57:09

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

oops...a little issue.

P.S. hosts.head does <i>not</i>get mangled. Happy bug stomping all!

Version 0.10.2 (09.08.2012)
*fixed issue with 0.0.0.0 address not working
*cleaned up logging output

Known issues:
*when calculating number of entries at end, mistakes hosts.head
    domains as redirect addresses


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#91 2012-08-10 05:54:10

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

Fixed up the bug.

Thanks. But now you broke the white.list processing. lol

Last edited by ontobelli (2012-08-10 13:22:53)

Offline

#92 2012-08-10 15:47:17

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.10.3 (10.08.2012)
*fixed white.list issue

HIGHLY ADVISED: The "optimization" feature seems to actually DECREASES performance. Until we figure out a way to wrap the lines to some reasonable length, it's not advisable to use it.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#93 2012-08-10 18:40:17

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

HIGHLY ADVISED: The "optimization" feature seems to actually DECREASE performance. Until we figure out a way to wrap the lines to some reasonable length, it's not advisable to use it.

Thanks...
I've used about 8MB hosts files and have not seen any performance problems to date.
The so-called optimization decreases file size, but makes its management much more complicated.
I personally think it's not worth it.

Offline

#94 2012-08-11 14:09:37

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@sadi. I agree, and it adds a lot of overhead to the hostsblock processing as well. I'm taking it out.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#95 2012-08-11 14:16:59

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.10.4 (11.08.2012)
*removed optimization code. Caused significant performance
    penalties
*fixed old issue with hostsblock-url
*replaced a few sed invocations with tr for slight     
    performance increase


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#96 2012-08-13 10:28:54

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

A few improvements in /etc/hostsblock/rc.conf

postprocess(){
    systemctl restart dnsmasq   # For systemd dnsmasq service
}

simplify URL

http://hosts-file.net/ base address is the same for
-------------------------------------------------------------------------------
http://hosts-file.net/download/hosts.zip
http://hosts-file.net/hphosts-partial.asp
http://hosts-file.net/ad_servers.asp
http://hosts-file.net/download/yahoo_servers.zip 

Blocks all Yahoo servers should be moved from recommend to optional lists.
No need the domain http://support.it-mate.co.uk

http://hostsfile.org/ base address is the same for
-----------------------------------------------------------------------------
http://hostsfile.org/Downloads/BadHosts.unx.zip
http://hostsfile.org/Downloads/hosts.txt (same as www.securemecca.com)

My suggestion is this

blocklists=(
## HIGHLY RECOMMENDED LISTS
'http://winhelp2002.mvps.org/hosts.zip'
'http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext'
'http://hosts-file.net/download/hosts.zip'
## RECOMMENDED LISTS
'http://hosts-file.net/hphosts-partial.asp'
'http://hosts-file.net/ad_servers.asp'
'http://hostsfile.org/Downloads/BadHosts.unx.zip'
'http://hostsfile.org/Downloads/hosts.txt'
'http://hostsfile.mine.nu/Hosts.zip'
'http://someonewhocares.org/hosts/hosts'
'http://sysctl.org/cameleon/hosts'
'http://www.ismeh.com/HOSTS'
'http://www.malwaredomainlist.com/hostslist/hosts.txt'
## OPTIONAL LISTS
#'http://abp.mozilla-hispano.org/nauscopio/hosts.zip'
#'http://hosts-file.net/download/yahoo_servers.zip'
#'http://rlwpx.free.fr/WPFF/htrc.7z'
#'http://rlwpx.free.fr/WPFF/hpub.7z'
#'http://rlwpx.free.fr/WPFF/hrsk.7z'	
#'http://rlwpx.free.fr/WPFF/hsex.7z'
#'http://rlwpx.free.fr/WPFF/hmis.7z'
## LISTS WITH REDIRECTIONS. Enable redirects="1" to enable, and also uncomment
#'http://rlwpx.free.fr/WPFF/hblc.7z'
#'http://www.modyouri.com/adblock_hosts/hosts'
#'http://www.securemecca.com/Downloads/hosts.txt'
)

Offline

#97 2012-08-13 14:21:35

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Good idea. On further inspection: 'http://hostsfile.org/Downloads/BadHosts.unx.zip' and 'http://hostsfile.org/Downloads/hosts.txt' are identical, and 'http://hosts-file.net/ad_servers.asp' is just a subset of 'http://hosts-file.net/download/hosts.zip'. I'll roll that into my latest update.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#98 2012-08-13 14:28:48

gaenserich
Member
From: Michigan
Registered: 2010-06-07
Posts: 119
Website

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

Version 0.10.5 (12.08.2012)
*updated rc.conf, removing redundant blocklists.
*fixed a few issues with hostsblock-urllist. Should now
    register more domains and filter out any garbage.


Check out hostsblock for system-wide ad- and malware-blocking.

Offline

#99 2012-08-13 14:33:54

sadi
Member
From: Istanbul
Registered: 2012-04-02
Posts: 51

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

@gaenserich: some users might just be content with 'http://hosts-file.net/ad_servers.asp' instead of the very large (and very "aggressive") 'http://hosts-file.net/download/hosts.zip', which contains some commercial sites frequently visited by people for shopping etc.
So I think it might be a good idea to leave these 2 options with informative comments.

Offline

#100 2012-08-13 14:55:18

ontobelli
Member
From: Mexico City
Registered: 2011-02-06
Posts: 127

Re: hostsblock: cli utility/cronscript to automate hostsfile adblocking

gaenserich wrote:

Not always.

hosts.zip is a monthly list

ad_servers.asp and hphosts-partial.asp are dynamically generated with the latest additions and could change many times in a single day.

In my opinion is good to include the 3 lists.  Let the script do the job wink

***

By the way 'http://www.malwaredomainlist.com/hostslist/hosts.txt' should be in the HIGHLY RECOMMENDED LISTS section.

Offline

Board footer

Powered by FluxBB