Arch Linux

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

(SOLVED) iptables No Longer Restores Rules on Boot

Update
After additional troubleshooting, it seems I misdiagnosed the problem. iptables-restore is working as expected, but the problem is with connection tracking. I usually reboot the system by scheduling a shutdown with “shutdown -r” over SSH. If I then try to communicate with the system from the same IP address, the connection will be refused. If I communicate with it from any other IP address, even if it’s just an ICMP echo packet, then everything goes back to normal.

I configured tcpdump to start on boot, and the packet capture confirms the system is receiving traffic that is then rejecting, so this is not a problem with the VPS provider (Linode). Strangely, when I configured iptables to log all dropped and rejected packets, it failed to log any of the rejected packets from my system (although it did log rejections and drops from other sources).

Original
On 2018-11-09, I ran pacman, installed all available updates, and rebooted. When my system came up, I noticed I couldn’t connect to it via SSH. When I logged in via other means, I noticed the iptables rules were back to their default values. After running iptables-restore and ip6tables-restore, access returned to normal, but this problem has recurred during every subsequent reboot. I have not changed the iptables rules since 2018-07-30, and I have not had this problem with this system (which has been in production since 2017), so I’m not certain what happened or where to look.

Updated packages:

[2018-11-09 07:55] [ALPM] upgraded m4 (1.4.18-1 -> 1.4.18-2)
[2018-11-09 07:55] [ALPM] upgraded diffutils (3.6-1 -> 3.6-2)
[2018-11-09 07:55] [ALPM] upgraded gmp (6.1.2-1 -> 6.1.2-2)
[2018-11-09 07:55] [ALPM] upgraded autoconf (2.69-4 -> 2.69-5)
[2018-11-09 07:55] [ALPM] upgraded bison (3.1-1 -> 3.2-1)
[2018-11-09 07:55] [ALPM] upgraded gdbm (1.18-1 -> 1.18.1-1)
[2018-11-09 07:55] [ALPM] upgraded libffi (3.2.1-2 -> 3.2.1-3)
[2018-11-09 07:55] [ALPM] upgraded libutil-linux (2.32.1-2 -> 2.33-2)
[2018-11-09 07:55] [ALPM] upgraded python-requests (2.20.0-1 -> 2.20.1-1)
[2018-11-09 07:55] [ALPM] upgraded python-acme (0.27.1-1 -> 0.28.0-1)
[2018-11-09 07:55] [ALPM] upgraded findutils (4.6.0-2 -> 4.6.0-4)
[2018-11-09 07:55] [ALPM] upgraded libcap (2.25-1 -> 2.25-2)
[2018-11-09 07:55] [ALPM] upgraded certbot (0.27.1-1 -> 0.28.0-1)
[2018-11-09 07:55] [ALPM] upgraded cracklib (2.9.6-1 -> 2.9.6-3)
[2018-11-09 07:55] [ALPM] upgraded flex (2.6.4-1 -> 2.6.4-2)
[2018-11-09 07:55] [ALPM] installed libcroco (0.6.12+4+g9ad7287-1)
[2018-11-09 07:55] [ALPM] upgraded gettext (0.19.8.1-2 -> 0.19.8.1-3)
[2018-11-09 07:55] [ALPM] upgraded grep (3.1-1 -> 3.1-2)
[2018-11-09 07:55] [ALPM] upgraded groff (1.22.3-7 -> 1.22.3-8)
[2018-11-09 07:55] [ALPM] upgraded gzip (1.9-1 -> 1.9-2)
[2018-11-09 07:55] [ALPM] upgraded inetutils (1.9.4-5 -> 1.9.4-6)
[2018-11-09 07:55] [ALPM] upgraded kbd (2.0.4-1 -> 2.0.4-2)
[2018-11-09 07:55] [ALPM] upgraded libaio (0.3.111-1 -> 0.3.111-2)
[2018-11-09 07:55] [ALPM] upgraded libevent (2.1.8-1 -> 2.1.8-2)
[2018-11-09 07:55] [ALPM] upgraded lzo (2.10-1 -> 2.10-2)
[2018-11-09 07:55] [ALPM] upgraded libmariadbclient (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded libmnl (1.0.4-1 -> 1.0.4-2)
[2018-11-09 07:55] [ALPM] upgraded licenses (20171006-1 -> 20181104-1)
[2018-11-09 07:55] [ALPM] upgraded tar (1.30-1 -> 1.30-2)
[2018-11-09 07:55] [ALPM] upgraded make (4.2.1-2 -> 4.2.1-3)
[2018-11-09 07:55] [ALPM] upgraded mariadb-clients (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded mariadb (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded patch (2.7.6-3 -> 2.7.6-4)
[2018-11-09 07:55] [ALPM] upgraded util-linux (2.32.1-2 -> 2.33-2)
[2018-11-09 07:55] [ALPM] upgraded reiserfsprogs (3.6.27-1 -> 3.6.27-2)
[2018-11-09 07:55] [ALPM] upgraded sqlite (3.25.2-1 -> 3.25.3-1)
[2018-11-09 07:55] [ALPM] upgraded traceroute (2.1.0-1 -> 2.1.0-2)
[2018-11-09 07:55] [ALPM] upgraded zstd (1.3.6-1 -> 1.3.7-1)

iptables version:

iptables v1.8.0 (legacy)

audit records showing iptables-restore and ip6tables-restore are running during boot (although I don’t know why they’re generating 5 records each time; this also happens if I manually run the commands):

Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd2b50 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd3240 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd3ad0 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd4530 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cdb840 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=iptables comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194ddf0 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tt
y=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194e360 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tt
y=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194ff30 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f1950460 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f1956340 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)

Last edited by AstroPig7 (2018-11-27 18:19:20)

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

Ah! I almost forgot to say that iptables.service and ip6tables.service are pointing to the correct rule files:

From iptables.service:

ExecStart=/usr/bin/iptables-restore /etc/iptables/iptables.rules
ExecReload=/usr/bin/iptables-restore /etc/iptables/iptables.rules

From ip6tables.service:

ExecStart=/usr/bin/ip6tables-restore /etc/iptables/ip6tables.rules
ExecReload=/usr/bin/ip6tables-restore /etc/iptables/ip6tables.rules

Haller

Member

Registered: 2018-04-08

Posts: 45

Re: (SOLVED) iptables No Longer Restores Rules on Boot

What does "systemctl status iptables" say?
And can you post the output of "journalctl -u iptables"?

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

What does "systemctl status iptables" say?
And can you post the output of "journalctl -u iptables"?

systemctl status iptables:

iptables.service - Packet Filtering Framework
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2018-11-16 14:17:42 CST; 1h 18min ago
  Process: 436 ExecStart=/usr/bin/iptables-restore /etc/iptables/iptables.rules (code=exited, status=0/SUCCESS)
 Main PID: 436 (code=exited, status=0/SUCCESS)

Nov 16 14:17:42 archlinux systemd[1]: Starting Packet Filtering Framework...
Nov 16 14:17:42 archlinux systemd[1]: Started Packet Filtering Framework.

journalctl -u iptables (since the last reboot):

Nov 16 14:17:42 archlinux systemd[1]: Starting Packet Filtering Framework...
Nov 16 14:17:42 archlinux systemd[1]: Started Packet Filtering Framework.

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

As a test, I added an ExecStartPost value to iptables.service that makes it save the current rules to a file (not the same file it loads the rules from). The output confirms that iptables is successfully loading the rules from file, which implies that something is resetting the rules after iptables loads. Unfortunately, whatever is doing it is not leaving any evidence in the systemd journal.

amish

Member

Registered: 2014-05-10

Posts: 470

Re: (SOLVED) iptables No Longer Restores Rules on Boot

grep 'ip6\?tables' /etc/systemd/system/*/*.service
grep 'ip6\?tables' /usr/lib/systemd/system/*/*.service

Other than ip[6]tables.service Anything else calling iptables?

Last edited by amish (2018-11-17 06:49:05)

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

Problem changed.

I configured Linux Auditing System to track all system calls of type a3=0x40 (i.e., the syscalls made to iptables), and it recorded nothing other than the usual service startup and my own reload of the iptables rules. By accident, I noticed that if I log in as myself directly (i.e., not through SSH) and don’t run iptables-restore, then I’m suddenly able to connect using SSH. I then checked

iptables -nvL

and found all of my rules were in place. I think I misread the rules table when this problem first started (Linode’s backend console is weird at first), and I now think this is actually related to connection tracking. I changed my SSH rule from

-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j IN_SSH

to

-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j IN_SSH

but that didn’t fix the problem. The only rule in the IN_SSH chain is

-A IN_SSH -m recent --set --name sshbf --mask 255.255.255.255 --rsource -j ACCEPT

I should note that my physical Arch Linux system does not have this problem, and it only uses

-A TCP -p tcp -m tcp --dport 22 -j ACCEPT

(i.e., there is no IN_SSH chain or connection tracking).

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

I fixed the problem by switching from Linode's custom kernel to the upstream kernel supplied through pacman.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,772

Re: (SOLVED) iptables No Longer Restores Rules on Boot

*HeadDesk*

Mayhaps you should have mentioned Linode sometime prior to post #7 ?

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

AstroPig7

Member

Registered: 2018-11-16

Posts: 8

Re: (SOLVED) iptables No Longer Restores Rules on Boot

I’ll take that headdesk and double it. I’ve been working in IT so long that my users’ ability to omit the most critical piece of information seems to have rubbed off on me.

Last edited by AstroPig7 (2018-11-28 03:20:53)