Strange masquerading behavior, how to troubleshoot?

rklingsten · 2009-11-18 17:44:51

Hi folks -

I've done this same setup dozens of times with no problems, but this time something very strange is happening. Any suggestions on where to look will be most appreciated.

I've got an Arch machine (up to date) connected to the Internet via DSL; the DSL modem is connected to eth0 and I'm running ppp. The local LAN is connected to eth1. I've got the ip_forward set to 1 in /proc, and set up as:

------------------------------------------------------------

eth0      Link encap:Ethernet  HWaddr 00:1C:C0:7D:B9:CE  
          inet6 addr: fe80::21c:c0ff:fe7d:b9ce/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:205042 errors:0 dropped:0 overruns:0 frame:0
          TX packets:169980 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:248601427 (237.0 Mb)  TX bytes:21764846 (20.7 Mb)
          Interrupt:27 Base address:0xc000 

eth1      Link encap:Ethernet  HWaddr 00:1B:21:2F:E2:84  
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:fe2f:e284/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:175145 errors:0 dropped:0 overruns:0 frame:0
          TX packets:203011 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21295740 (20.3 Mb)  TX bytes:245089951 (233.7 Mb)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:276 (276.0 b)  TX bytes:276 (276.0 b)

ppp0      Link encap:Point-to-Point Protocol  
          inet addr:75.45.205.10  P-t-P:75.45.207.254  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:202705 errors:0 dropped:0 overruns:0 frame:0
          TX packets:167631 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3 
          RX bytes:243993889 (232.6 Mb)  TX bytes:18004911 (17.1 Mb)

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
75.45.207.254   0.0.0.0         255.255.255.255 UH        0 0          0 ppp0
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 ppp0

-----------------------------------------------------------------------

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     any     anywhere             anywhere            
    1   229 ACCEPT     all  --  eth1   any     anywhere             anywhere            
   37  2788 ACCEPT     all  --  ppp0   any     anywhere             anywhere            state RELATED,ESTABLISHED 
    0     0 ACCEPT     tcp  --  ppp0   any     anywhere             anywhere            tcp dpt:ssh 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 19 packets, 2172 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain PREROUTING (policy ACCEPT 1 packets, 229 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    ppp0    anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

-----------------------------------------------------------------------

I am also using dnsmasq on the gateway and have DNS pointed to OpenDNS (208.67.222.222 & 220.220.) The LAN computers all get DHCP from the gateway as well as point to the gateway as their DNS server.

The trouble I am having is that various websites (i.e. www.microsoft.com, www.msn.com, www.ramsey.com) on the LAN computers fail to come up. The browsers (Firefox and lynx on Linux, IE and Firefox on Windows) show in their status bars "Waiting for response" and ultimately fail with a timeout. The majority of sites from the same computers work just fine and come up with no problems.

Also, using lynx on the gateway machine itself, www.microsoft.com and the others all come up right away. So something is happening with the packet forwarding. I've power cycled the LAN switch in case it was some weird problem with that, didn't help.

Here's a tcpdump -v port www from a LAN machine, where www.microsoft.com won't come up, this is the entire transaction after typing in lynx "G" and "http://www.microsoft.com":

12:37:27.664840 IP office.local.60955 > 65.55.21.250.www: S 1712443788:1712443788(0) win 5840 <mss 1460,sackOK,timestamp 297882074 0,nop,wscale 5>
12:37:27.737455 IP 65.55.21.250.www > office.local.60955: S 3115800200:3115800200(0) ack 1712443789 win 8190 <mss 1460>
12:37:27.737509 IP office.local.60955 > 65.55.21.250.www: . ack 1 win 5840
12:37:27.738034 IP office.local.60955 > 65.55.21.250.www: P 1:227(226) ack 1 win 5840
12:37:27.819376 IP 65.55.21.250.www > office.local.60955: FP 1:538(537) ack 227 win 63962
12:37:27.819401 IP office.local.60955 > 65.55.21.250.www: . ack 539 win 6444
12:37:27.820280 IP office.local.60955 > 65.55.21.250.www: F 227:227(0) ack 539 win 6444
12:37:27.892660 IP 65.55.21.250.www > office.local.60955: . ack 228 win 63962
12:37:31.837692 IP office.local.60956 > 65.55.21.250.www: S 1774824928:1774824928(0) win 5840 <mss 1460,sackOK,timestamp 297883118 0,nop,wscale 5>
12:37:31.910079 IP 65.55.21.250.www > office.local.60956: S 2783839720:2783839720(0) ack 1774824929 win 8190 <mss 1460>
12:37:31.910095 IP office.local.60956 > 65.55.21.250.www: . ack 1 win 5840
12:37:31.910554 IP office.local.60956 > 65.55.21.250.www: P 1:245(244) ack 1 win 5840
12:37:34.909435 IP office.local.60956 > 65.55.21.250.www: P 1:245(244) ack 1 win 5840
12:37:34.984171 IP 65.55.21.250.www > office.local.60956: . ack 245 win 63034
12:37:51.069170 IP 65.55.21.250.www > office.local.60956: R 2783841181:2783841181(0) win 9700
12:37:51.085398 IP office.local.60957 > 65.55.21.250.www: S 2073536154:2073536154(0) win 5840 <mss 1460,sackOK,timestamp 297887929 0,nop,wscale 5>
12:37:51.157796 IP 65.55.21.250.www > office.local.60957: S 4286192132:4286192132(0) ack 2073536155 win 8190 <mss 1460>
12:37:51.157814 IP office.local.60957 > 65.55.21.250.www: . ack 1 win 5840
12:37:51.158330 IP office.local.60957 > 65.55.21.250.www: P 1:28(27) ack 1 win 5840
12:37:51.231956 IP 65.55.21.250.www > office.local.60957: . ack 28 win 8163

And at this point, the browser is just sitting there with "Waiting for response" and then finally times out and fails.

Any ideas or suggestions? Thanks for any help!

Rob K

fukawi2 · 2009-11-18 21:52:22

The only odd thing I can see in what you've got is this:

rklingsten wrote:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
75.45.207.254   0.0.0.0         255.255.255.255 UH        0 0          0 ppp0
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 ppp0

There's no actual default gateway set, which suggests your ISP is doing some nasty, dirty ARP proxying. What is the output of `arp -n` after the box has been up for a while?

Other than that I can't see anything wrong. If your public IP address is static, perhaps try using a SNAT instead of MASQUERADE?

Last edited by fukawi2 (2009-11-18 21:52:38)

rklingsten · 2009-11-19 19:34:11

Doesn't 0.0.0.0/0 mean the same thing as default route? /0 would be the shortest match possible. And arp -n just shows the couple of LAN computers, as expected to be there, nothing funny.

thanks -

Rob K

Last edited by rklingsten (2009-11-19 19:34:29)

fukawi2 · 2009-11-19 21:54:30

rklingsten wrote:

Doesn't 0.0.0.0/0 mean the same thing as default route? /0 would be the shortest match possible.

Yes, but the gateway column is also 0.0.0.0 where I thought it should have an IP address at the other end of the PPP tunnel. I stand corrected though after checking a box I know is using PPP:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.20.30.248    0.0.0.0         255.255.255.248 U     0      0        0 eth2
10.20.30.0      0.0.0.0         255.255.255.192 U     0      0        0 eth1
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 ppp0

IOW, I can't see anything wrong I'm afraid. Perhaps further analysis of the tcpdump capture using Wireshark might show some errors within the outgoing packets or something...

Did you try a SNAT or do you have a dynamic address?

ataraxia · 2009-11-19 22:29:31

I see that your uplink has a smaller MTU than your LAN link. From http://wiki.archlinux.org/index.php/Sim … wall_HOWTO:

Some networks and servers block ICMP messages and are preventing path MTU detection. If your outgoing interface's MTU is lower than the MTU on the local network (like with PPPoE connections), this may prevent you from communicating with such servers. The first rule works around this problem, at least for TCP connections:
# iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

rklingsten · 2009-11-30 18:51:16

I gave up, since I was getting nonstop complaint about it not working. Thanks for the suggestions, I'll bet the MTU issue suggested was it. I replaced it with a pfsense installation and it now works perfectly. We'll see if I get the opportunity to try it out again with the MTU fix suggested.

thanks again for helping!

Rob K

Arch Linux

#1 2009-11-18 17:44:51

Strange masquerading behavior, how to troubleshoot?

#2 2009-11-18 21:52:22

Re: Strange masquerading behavior, how to troubleshoot?

#3 2009-11-19 19:34:11

Re: Strange masquerading behavior, how to troubleshoot?

#4 2009-11-19 21:54:30

Re: Strange masquerading behavior, how to troubleshoot?

#5 2009-11-19 22:29:31

Re: Strange masquerading behavior, how to troubleshoot?

#6 2009-11-30 18:51:16

Re: Strange masquerading behavior, how to troubleshoot?

Board footer