Squid and browser on the same machine - All work and no play [SOLVED]

MonkeyBoy · 2017-11-26 23:04:33

[Solution to be found in last post]

Hi. New to the forum.

Have been trying to get squid as transparent proxy working on my laptop, can't really wrap my head around what ip(:s) to use where when localhost comes into play, access.log never become populated.

Have tried to set /etc/environment to localhost and ip provided by the router, but to no avail

HTTP_PROXY="http://127.0.0.1:3128/"

Have tried 127.0.0.1 and the ip provided to the laptop in squid.conf and iptables.rules. Have restarted squid after every change. To no avail

Have I gone 'blind' after all my tries? Please help a poor newbie.

My squid.conf

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8           # RFC1918 possible internal network
acl localnet src 172.16.0.0/12     # RFC1918 possible internal network
acl localnet src 192.168.43.0/24 # RFC1918 possible internal network
acl localnet src fc00::/7               # RFC 4193 local private network range
acl localnet src fe80::/10             # RFC 4291 link-local (directly plugged) machines
acl localned src 127.0.0.1           # localhost, found a blogpost on serverfault suggesting it

acl SSL_ports port 443
acl Safe_ports port 80                 # http
acl Safe_ports port 21                # ftp
acl Safe_ports port 443               # https
acl Safe_ports port 70                 # gopher
acl Safe_ports port 210               # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280                # http-mgmt
acl Safe_ports port 488               # gss-http
acl Safe_ports port 591               # filemaker
acl Safe_ports port 777               # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
# And finally deny all other access to this proxy
http_access allow all        # For the time being

# Squid normally listens to port 3128
http_port 3128 intercept        # According to Arch wiki
http_port 3129

# host name, according to Arch wiki
visible_hostname squiddifar

# shutdown time, according to Arch wiki
shutdown_lifetime 1 seconds

# cache memory, according to Arch wiki
cache_mem 256 MB

# Maximal object size, according to Arch wiki
maximum_object_size 16 MB

# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/cache/squid 1024 16 256

# Visible host name, according to Arch wiki
visible_hostname Squiddifar

# Leave coredumps in the first cache dir
coredump_dir /var/cache/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:               1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?)      0     0%      0
refresh_pattern .                           0     20%     4320

/etc/iptables/iptables.rules

 Generated by iptables-save v1.6.1 on Sun Nov 26 22:56:08 2017
*filter
:INPUT ACCEPT [5022:1199674]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4238:668740]
COMMIT
# Completed on Sun Nov 26 22:56:08 2017
# Generated by iptables-save v1.6.1 on Sun Nov 26 22:56:08 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [62:4252]
:POSTROUTING ACCEPT [72:4852]
-A OUTPUT -p tcp -m tcp --dport 80 -m owner --gid-owner 15 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 80 -j DNAT --to-destination 127.0.0.1$
COMMIT
# Completed on Sun Nov 26 22:56:08 2017

Last edited by MonkeyBoy (2017-12-03 22:36:07)

nomorewindows · 2017-11-27 03:18:11

For same machine, you don't really need iptables or intercept, and polipo might be easier to setup and use.
In order to clean up cache in squid, you'd need to use squidpurge, not that it differs that much in polipo...
Using a proxy with https is kind of pointless though.
http_proxy=http://localhost:3128 is probably sufficient although I have static address on my interface so I use that and it would probably like that better than localhost or 127.0.0.1...

Last edited by nomorewindows (2017-11-27 03:18:33)

MonkeyBoy · 2017-11-27 20:05:18

Saw that I missed to give the link to Serverfault in the opening of the thread.

@nomorewindows

I worked along this config example from the Squid homepage

Tried set SQUIDIP to both 127.0.0.1 and ip given by router

NP: for most non-Windows boxes setting the http_proxy environment variable (http_proxy="http://SQUIDIP:3128/") is a preferred alternative to the below interception.

(Every change in conf files or the like was) followed by

monkeyboy@bananatree ~> sudo systemctl restart squid
monkeyboy@bananatree ~> sudo tail -f /var/log/squid/access.log

access.log not populated. I commented out http_proxy in /etc/environment and the I tried this

# normal transparent proxy
iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 3128

# handle connections on the same box (SQUIDIP is a loopback instance)
gid=`id -g proxy`
iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --gid-owner $gid -j ACCEPT
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination SQUIDIP:3128

I tried SQUIDIP to both 127.0.0.1 and ip given to my machine.

This is the iptabels.rules

# Generated by iptables-save v1.6.1 on Mon Nov 27 20:16:27 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [1:63]
:POSTROUTING ACCEPT [5:303]
-A PREROUTING -i wlp3s0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128
-A OUTPUT -p tcp -m tcp --dport 80 -m owner --gid-owner 15 -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.43.206:3128
COMMIT
# Completed on Mon Nov 27 20:16:27 2017

And that's where I am right now. Maybe I should go for Polipo, but I thought I would give this a try as I have already started, and apprently people get it working.

If I understand you right I should squidpurge, no? But I have nothing to purge, right? access-log is not populated and I cannot find anything at /var/cache/squid but the folders and nothing in the files.

And then you suggest I would set env http_proxy to the ip address I am provided by my router, yes? (In my case my smart phone, due to circumstances I am on a 4G connection and want to save quota, hence catch server) Had already tried that in thread opening.

I will fiddle around a little more to see if I can manage. Thanks for your effort though - Cheers!

circleface · 2017-11-27 21:11:53

I noticed you have a typo in the config:

acl localned src 127.0.0.1 # localhost, found a blogpost on serverfault suggesting it

That should be localnet. Not sure that will fix anything, but it can't hurt to get it right.

http_access deny CONNECT !SSL_ports

Is that right? It seems to me that that blocks all ports but the SSL ones, which don't work with a proxy anyway.

Edit: spelling.

Last edited by circleface (2017-11-27 22:46:12)

MonkeyBoy · 2017-11-27 21:44:48

@ circle face - well spotted! I will def change that. And try it all again Could very well be the culprit.

Reg. the SSL ports. That setting comes with the install. Minimum config sts.

Again, thanks for seeing where I had gone blind - Cheers!

MonkeyBoy · 2017-11-27 23:08:13

@circleface - Wee : )

Thanks a bunch, correcting that typo you spotted populated access.log.

Now, there is a slight problem: access.log is populated but it is all MISS. Seems like MISS meaning that it is not getting anything from the local cache. tail -f gets a line if I open a page, but things are not really flowing as they should - right? So nothing is really cached, right?
Posting my access.log

1511812307.141    137 192.168.43.206 TCP_MISS/200 473 GET http://manpages.ubuntu.com/assets/light/images/body_bg.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.141    126 192.168.43.206 TCP_MISS/200 1958 GET http://manpages.ubuntu.com/assets/light/images/search_submit_bg_2.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.299    122 192.168.43.206 TCP_MISS/200 863 GET http://manpages.ubuntu.com/img/printer.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.302    124 192.168.43.206 TCP_MISS/200 1890 GET http://manpages.ubuntu.com/assets/light/images/footer_logo.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.306    125 192.168.43.206 TCP_MISS/200 575 GET http://manpages.ubuntu.com/assets/light/images/subnav_divider.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.430    116 192.168.43.206 TCP_MISS/200 530 GET http://manpages.ubuntu.com/assets/light/images/bullet.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.448    127 192.168.43.206 TCP_MISS/200 1513 GET http://manpages.ubuntu.com/assets/light/images/favicon.ico - ORIGINAL_DST/91.189.95.15 image/vnd.microsoft.icon
1511813484.343    326 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.221.114 text/html
1511818814.444    370 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.222.114 text/html
1511822871.171   1047 192.168.43.206 TCP_MISS/302 665 GET http://wiki.squid-cache.org/SquidFaq/SquidLogs - ORIGINAL_DST/104.130.201.120 text/html

nomorewindows · 2017-11-28 06:33:54

MonkeyBoy wrote:

@circleface - Wee : )

Thanks a bunch, correcting that typo you spotted populated access.log.

Now, there is a slight problem: access.log is populated but it is all MISS. Seems like MISS meaning that it is not getting anything from the local cache. tail -f gets a line if I open a page, but things are not really flowing as they should - right? So nothing is really cached, right?
Posting my access.log

1511812307.141    137 192.168.43.206 TCP_MISS/200 473 GET http://manpages.ubuntu.com/assets/light/images/body_bg.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.141    126 192.168.43.206 TCP_MISS/200 1958 GET http://manpages.ubuntu.com/assets/light/images/search_submit_bg_2.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.299    122 192.168.43.206 TCP_MISS/200 863 GET http://manpages.ubuntu.com/img/printer.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.302    124 192.168.43.206 TCP_MISS/200 1890 GET http://manpages.ubuntu.com/assets/light/images/footer_logo.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.306    125 192.168.43.206 TCP_MISS/200 575 GET http://manpages.ubuntu.com/assets/light/images/subnav_divider.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.430    116 192.168.43.206 TCP_MISS/200 530 GET http://manpages.ubuntu.com/assets/light/images/bullet.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.448    127 192.168.43.206 TCP_MISS/200 1513 GET http://manpages.ubuntu.com/assets/light/images/favicon.ico - ORIGINAL_DST/91.189.95.15 image/vnd.microsoft.icon
1511813484.343    326 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.221.114 text/html
1511818814.444    370 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.222.114 text/html
1511822871.171   1047 192.168.43.206 TCP_MISS/302 665 GET http://wiki.squid-cache.org/SquidFaq/SquidLogs - ORIGINAL_DST/104.130.201.120 text/html

It's going to be a miss on the first run, until you come back later will there be a hit...

MonkeyBoy · 2017-11-28 19:00:51

@nomorewindows

Something is weird here. I start the computer, Squid is enabled so it runs from boot, I check the "systemctl status squid" and it was active. So I open my browser, then "tail -f access.log"

monkeyboy@bananatree ~> sudo tail -f /var/log/squid/access.log
1511812307.299    122 192.168.43.206 TCP_MISS/200 863 GET http://manpages.ubuntu.com/img/printer.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.302    124 192.168.43.206 TCP_MISS/200 1890 GET http://manpages.ubuntu.com/assets/light/images/footer_logo.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.306    125 192.168.43.206 TCP_MISS/200 575 GET http://manpages.ubuntu.com/assets/light/images/subnav_divider.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.430    116 192.168.43.206 TCP_MISS/200 530 GET http://manpages.ubuntu.com/assets/light/images/bullet.png - ORIGINAL_DST/91.189.95.15 image/png
1511812307.448    127 192.168.43.206 TCP_MISS/200 1513 GET http://manpages.ubuntu.com/assets/light/images/favicon.ico - ORIGINAL_DST/91.189.95.15 image/vnd.microsoft.icon
1511813484.343    326 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.221.114 text/html
1511818814.444    370 192.168.43.206 TCP_MISS/200 604 GET http://check.googlezip.net/connect - ORIGINAL_DST/173.194.222.114 text/html
1511822871.171   1047 192.168.43.206 TCP_MISS/302 665 GET http://wiki.squid-cache.org/SquidFaq/SquidLogs - ORIGINAL_DST/104.130.201.120 text/html
1511823561.148   1802 192.168.43.206 TCP_MISS/301 647 GET http://tools.ietf.org/rfc/rfc1945 - ORIGINAL_DST/4.31.198.62 text/html
1511823681.187    348 192.168.43.206 TCP_MISS/301 294 GET http://dn.se/ - ORIGINAL_DST/52.28.224.144 -

So, OK, since yesterday there is two more lines, a newspaper I have on my tabs constantly and a document that

 https://wiki.squid-cache.org/SquidFaq/SquidLogs#access.log

refers to. But all lines are MISS, even SquidLogs. And then - should it not be a more or less a constant flow of lines when running "tail -f access.log" and I open tabs etc. in the browser?

Further I check "iptables-save | less" and that renders in zilch, though the "/etc/iptables/iptables.rules" is intact of course. The iptables.rules is what make the machinations with iptables permanent, no? So why "iptables-save" has nothing to show for?

Confusion in abundance over here.

Last edited by MonkeyBoy (2017-11-28 19:30:09)

MonkeyBoy · 2017-12-03 22:23:14

Solution and reason to fail when I set up a squid web proxy and web browser on the same machine (my laptop)

I followed, in general, install and configuration instructions of Arch wiki#Squid

Then I applied the instructions of Transparent web proxy with two extra lines of iptables configuration from the Squid wiki.With root privileges, like this:

##For a standard transparent proxy - according to Squid wiki
~> iptables -t nat -F  ## Clears the table for a blank slate
~> iptables -t nat -A PREROUTING -p tcp -i myPublicInterface --dport 80 -j REDIRECT --to-port 3128
## myPublicInterface being the physical interface with the public ip-address of my laptop. E.g. eth0, wlp3s0, etc.
 
## To handle connections on the same box (SQUIDIP is a loopback instance)
~> gid=`id -g proxy`
~> iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --gid-owner $gid -j ACCEPT
~> iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination SQUIDIP:3128
~> iptables-save > /etc/iptables/iptables.rules
~> systemctl restart iptables
## SQUIDIP being my public ip address. E.g. 192.168.1.234

Now, my mistake, reason to fail and a couple of hours lost to finding out the error of things was to have a data saving extension on my (Chromium) web browser. This gave an erratic flow of the log at /var/log/squid/access.log and sent me off on a wild goose hunt fiddling around with ip: in /etc/iptables/iptables.rules, trying again with iptables commandos, sometimes forgetting to restart iptables, so on and so forth, etc, und so weiter.

Children, don't do what I have done. Instead follow above instructions and make sure to get rid on any web caching functions operating before Squid.

Last edited by MonkeyBoy (2017-12-03 22:39:48)

Arch Linux

#1 2017-11-26 23:04:33

Squid and browser on the same machine - All work and no play [SOLVED]

#2 2017-11-27 03:18:11

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#3 2017-11-27 20:05:18

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#4 2017-11-27 21:11:53

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#5 2017-11-27 21:44:48

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#6 2017-11-27 23:08:13

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#7 2017-11-28 06:33:54

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#8 2017-11-28 19:00:51

Re: Squid and browser on the same machine - All work and no play [SOLVED]

#9 2017-12-03 22:23:14

Re: Squid and browser on the same machine - All work and no play [SOLVED]

Board footer