DNS (rogue?) inbound packets blocked?

Rexilion · 2014-02-28 14:14:48

I use opendns as our DNS. Both directly with resolv.conf and indirectly through unbound. Both type of machines often return entries like this:

firewall wrote:

feb 28 10:46:26 delta kernel: bpkt raw out: IN= OUT=enp0s18 SRC=10.32.32.65 DST=208.67.222.222 LEN=109 TOS=0x08 PREC=0xC0 TTL=64 ID=9378 PROTO=ICMP TYPE=3 CODE=3 [SRC=208.67.222.222 DST=10.32.32.65 LEN=81 TOS=0x08 PREC=0x20 TTL=54 ID=0 DF PROTO=UDP SPT=53 DPT=22590 LEN=61]

I'm blocking outgoing ICMP packets to prevent portscans. The DNS servers (208.67.222.222 and 208.67.220.220) do this *a lot*. It's sometimes hard to reproduce and other times it's flooding my logs with this.

DNS resolving does not seem to be affected. I'm at a loss what is going on here. Could this be netfilter being to strict?

The machines running unbound have this more often than the machiens without.

x33a · 2014-02-28 18:15:19

Use wireshark to see what DNS data these packets are carrying.

Edit: Is all of this ICMP traffic? Maybe it's because of network problem.

Last edited by x33a (2014-02-28 18:19:07)

brebs · 2014-02-28 18:33:32

Rexilion wrote:

I'm blocking outgoing ICMP packets to prevent portscans.

Where is that stated as a sensible trick? I don't believe it's recommended.

Rexilion · 2014-02-28 18:53:32

Thanks for looking into this!

x33a wrote:

Use wireshark to see what DNS data these packets are carrying.
Edit: Is all of this ICMP traffic? Maybe it's because of network problem.

I briefly looked at it with tcpdump and compared the output of the firewall with it. It seems to happen at random. And most of the time only whenever I issue a new DNS request.

I had this with my old router and our shiny new equipment which is composed of a router dedicated as a modem and a 'proper' router in front of it.

What should I be looking for? They are all UDP packets containing DNS info.

brebs wrote:

Rexilion wrote:
I'm blocking outgoing ICMP packets to prevent portscans.
Where is that stated as a sensible trick? I don't believe it's recommended.

It works for UDP. I could drop the specific ICMP code/type combo's but one of these is 3/3 (dest unreachable, port unavailable). So unblocking these requests will get me exposed. For TCP I filter for certain flags on outgoing packets and block those. I combined that with cmark to check only new connections.

So, nothing new here.

It seems to happen more often with unbound than with direct DNS (final server in resolv.conf).

Last edited by Rexilion (2014-02-28 18:53:49)

brebs · 2014-02-28 19:41:22

Rexilion wrote:

It works for UDP. I could drop the specific ICMP code/type combo's but one of these is 3/3 (dest unreachable, port unavailable).

Your firewall should just DROP, at INPUT, those UDP packets. Thus no need to firewall your OUTPUT

Blocking ICMP can *cause* network problems.

Rexilion · 2014-02-28 19:48:39

brebs wrote:

Rexilion wrote:
It works for UDP. I could drop the specific ICMP code/type combo's but one of these is 3/3 (dest unreachable, port unavailable).
Your firewall should just DROP, at INPUT, those UDP packets. Thus no need to firewall your OUTPUT

I firewall output because it let's me check if it's a response to a closed port. If a service binds to a port (like nc) it will be automatically opened in the firewall. This is because I can differentiate between a response that says 'listening socket opened, go ahead' (allow) and 'no listening socket, stay stealth' (block).

UDP for example sends back UDP packets in case of a daemon listening to that port. If nothing is 'behind' the port, then the kernel responds with an ICMP packet. So, there will only be outbound traffic in case something needs to be done.

One could use the socket match for this in an INPUT table but I'm using this method as the socket method did not reliably work. I think it had issue's with 'kernel' sockets with NFS (not sure though).

brebs wrote:

Blocking ICMP can *cause* network problems.

Never heard of that one. Could be something to look into. However, it's kind of odd that only DNS is suffering from this.

I have extensive logging enabled (ratelimited of course) and only these DNS queries make no sense.

The rest is just noice from broadcast services, multicast services and the occassional attempt if an forwarded inbound connection to my tor server (which is dead since a long time). All the other messages, I can understand. Just not this one. Why now? Why from the DNS server? Why towards a closed port? It must have communicated with that port at some time since the router is sending it through (so it has a conntrack entry for that).

Last edited by Rexilion (2014-02-28 19:50:22)

brebs · 2014-02-28 20:12:47

Rexilion wrote:

If nothing is 'behind' the port, then the kernel responds with an ICMP packet.

The usual drop-by-default firewall configuration will DROP the incoming "portscan" packets first, thus no OUTPUT. So your firewall strategy is weird

A DNS server is probably receiving tons of weird/broken/malicious traffic itself - maybe someone's spoofing your IP address, and sending it requests, or maybe the DNS server is overloaded and having to drop/corrupt some of its connection-tracking.

Rexilion · 2014-02-28 21:40:25

brebs wrote:

Rexilion wrote:
If nothing is 'behind' the port, then the kernel responds with an ICMP packet.
The usual drop-by-default firewall configuration will DROP the incoming "portscan" packets first, thus no OUTPUT. So your firewall strategy is weird

Please do tell me how. Are you referring to '-m ctstate --state ESTABLISED,RELATED' combined with a DROP after it?

Yeah, that might work for your average NAT. But if you have servers (torrent, tor, http) listening then it won't. If the ports are open and the server is not running then a TCP RST or a ICMP is send back. With this, it's not.

brebs wrote:

A DNS server is probably receiving tons of weird/broken/malicious traffic itself - maybe someone's spoofing your IP address, and sending it requests, or maybe the DNS server is overloaded and having to drop/corrupt some of its connection-tracking.

Kind of far fetched. But worth a thought. I might look at my kernel logs whenever I have my laptop somewhere else.

brebs · 2014-02-28 21:51:42

Rexilion wrote:

and the server is not running

OK, but in that situation, the stopping of e.g. postfix *should* coincide with removing the associated ACCEPT rules from the INPUT chain of iptables. If you're so concerned about a portscan.

Gcool · 2014-03-01 10:13:19

What's probably happening here is that your dns request timed out for whatever reason, upon which the dns server sends an icmp type 3 packet (which is perfectly normal), which you are blocking.

All in all, nothing suspicious and/or much to worry about.

0strodamus · 2014-03-01 15:15:28

Are you using pdnsd? I noticed similar, if not the same, behavior which stopped when I switched to dnsmasq.

Rexilion · 2014-03-01 15:40:07

Gcool wrote:

What's probably happening here is that your dns request timed out for whatever reason, upon which the dns server sends an icmp type 3 packet (which is perfectly normal), which you are blocking.
All in all, nothing suspicious and/or much to worry about.

Yes, nothing *that* suspicious. But it was worrying because of the frequency. I assume you meant *my* DNS server? (the ICMP 3/3 is originating from my machine).

0strodamus wrote:

Are you using pdnsd? I noticed similar, if not the same, behavior which stopped when I switched to dnsmasq.

No, I use unbound and before that pdnsd. Unbound does the same as pdnsd but is still actively maintained.

Thank you for sharing your observations. I might try to be lazy and start playing with the timeout knobs in unbound.conf if any of those are relevant. Otherwise it's going to be dnsmasq (which I did not knew it was also a caching server).

Thanks!

0strodamus · 2014-03-01 18:15:26

Somehow I missed that you were using unbound in your first post. Probably, better that I did if my pdnsd observation helps you in any way.
I hope you get it figured out soon.

Rexilion · 2014-04-01 17:53:53

Ok, so I used systemD to multiplex (instantiate in systemD jargon) multiple DNS servers. My service file looks like this:

[Unit]
Description=A lightweight DHCP and caching DNS server
After=network.target
Documentation=man:dnsmasq(8)

[Service]
Type=simple
ExecStartPre=/usr/bin/dnsmasq -C /etc/dnsmasq/%I.conf --test
ExecStart=/usr/bin/dnsmasq -C /etc/dnsmasq/%I.conf -k --user=dnsmasq --pid-file
ExecReload=/bin/kill -HUP $MAINPID
Restart=always

[Install]
WantedBy=multi-user.target

I called it dnsmasq@.service and it's almost fully copied from the /usr/lib/systemd/system directory.

Then I created my main.conf (along with some custom .conf files) in /etc/dnsmasq.

To start the main DNS server with configuration file main.conf I simply do:

systemd start dnsmasq@main.service

It works great so far. Time will tell if these odd packets will come bouncing back. Using dnsmasq is a really nice idea. It has more options that I can use than unbound. For example all-servers and cache-size.

Will keep u gentlemen posted!

Rexilion · 2014-04-02 09:23:39

4,890,6935101546,-;bpkt raw out: IN= OUT=wlan0 SRC=10.32.32.67 DST=208.67.222.222 LEN=152 TOS=0x08 PREC=0xC0 TTL=64 ID=45850 PROTO=ICMP TYPE=3 CODE=3 [SRC=208.67.222.222 DST=10.32.32.67 LEN=124 TOS=0x08 PREC=0x20 TTL=55 ID=0 DF PROTO=UDP SPT=53 DPT=27408 LEN=104 ]

I'm starting to think it's my firewall. Maybe it's because I have forwarding enabled?

I'm steel keeping dnsmasq, it's neat.

Arch Linux

#1 2014-02-28 14:14:48

DNS (rogue?) inbound packets blocked?

#2 2014-02-28 18:15:19

Re: DNS (rogue?) inbound packets blocked?

#3 2014-02-28 18:33:32

Re: DNS (rogue?) inbound packets blocked?

#4 2014-02-28 18:53:32

Re: DNS (rogue?) inbound packets blocked?

#5 2014-02-28 19:41:22

Re: DNS (rogue?) inbound packets blocked?

#6 2014-02-28 19:48:39

Re: DNS (rogue?) inbound packets blocked?

#7 2014-02-28 20:12:47

Re: DNS (rogue?) inbound packets blocked?

#8 2014-02-28 21:40:25

Re: DNS (rogue?) inbound packets blocked?

#9 2014-02-28 21:51:42

Re: DNS (rogue?) inbound packets blocked?

#10 2014-03-01 10:13:19

Re: DNS (rogue?) inbound packets blocked?

#11 2014-03-01 15:15:28

Re: DNS (rogue?) inbound packets blocked?

#12 2014-03-01 15:40:07

Re: DNS (rogue?) inbound packets blocked?

#13 2014-03-01 18:15:26

Re: DNS (rogue?) inbound packets blocked?

#14 2014-04-01 17:53:53

Re: DNS (rogue?) inbound packets blocked?

#15 2014-04-02 09:23:39

Re: DNS (rogue?) inbound packets blocked?

Board footer