Mayhaps you should have mentioned Linode sometime prior to post #7 ?
]]>I configured Linux Auditing System to track all system calls of type a3=0x40 (i.e., the syscalls made to iptables), and it recorded nothing other than the usual service startup and my own reload of the iptables rules. By accident, I noticed that if I log in as myself directly (i.e., not through SSH) and don’t run iptables-restore, then I’m suddenly able to connect using SSH. I then checked
iptables -nvL
and found all of my rules were in place. I think I misread the rules table when this problem first started (Linode’s backend console is weird at first), and I now think this is actually related to connection tracking. I changed my SSH rule from
-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j IN_SSH
to
-A INPUT -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,ESTABLISHED -j IN_SSH
but that didn’t fix the problem. The only rule in the IN_SSH chain is
-A IN_SSH -m recent --set --name sshbf --mask 255.255.255.255 --rsource -j ACCEPT
I should note that my physical Arch Linux system does not have this problem, and it only uses
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT
(i.e., there is no IN_SSH chain or connection tracking).
]]>grep 'ip6\?tables' /etc/systemd/system/*/*.service
grep 'ip6\?tables' /usr/lib/systemd/system/*/*.service
Other than ip[6]tables.service Anything else calling iptables?
]]>What does "systemctl status iptables" say?
And can you post the output of "journalctl -u iptables"?
systemctl status iptables:
iptables.service - Packet Filtering Framework
Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; vendor preset: disabled)
Active: active (exited) since Fri 2018-11-16 14:17:42 CST; 1h 18min ago
Process: 436 ExecStart=/usr/bin/iptables-restore /etc/iptables/iptables.rules (code=exited, status=0/SUCCESS)
Main PID: 436 (code=exited, status=0/SUCCESS)
Nov 16 14:17:42 archlinux systemd[1]: Starting Packet Filtering Framework...
Nov 16 14:17:42 archlinux systemd[1]: Started Packet Filtering Framework.
journalctl -u iptables (since the last reboot):
Nov 16 14:17:42 archlinux systemd[1]: Starting Packet Filtering Framework...
Nov 16 14:17:42 archlinux systemd[1]: Started Packet Filtering Framework.
From iptables.service:
ExecStart=/usr/bin/iptables-restore /etc/iptables/iptables.rules
ExecReload=/usr/bin/iptables-restore /etc/iptables/iptables.rules
From ip6tables.service:
ExecStart=/usr/bin/ip6tables-restore /etc/iptables/ip6tables.rules
ExecReload=/usr/bin/ip6tables-restore /etc/iptables/ip6tables.rules
I configured tcpdump to start on boot, and the packet capture confirms the system is receiving traffic that is then rejecting, so this is not a problem with the VPS provider (Linode). Strangely, when I configured iptables to log all dropped and rejected packets, it failed to log any of the rejected packets from my system (although it did log rejections and drops from other sources).
Original
On 2018-11-09, I ran pacman, installed all available updates, and rebooted. When my system came up, I noticed I couldn’t connect to it via SSH. When I logged in via other means, I noticed the iptables rules were back to their default values. After running iptables-restore and ip6tables-restore, access returned to normal, but this problem has recurred during every subsequent reboot. I have not changed the iptables rules since 2018-07-30, and I have not had this problem with this system (which has been in production since 2017), so I’m not certain what happened or where to look.
Updated packages:
[2018-11-09 07:55] [ALPM] upgraded m4 (1.4.18-1 -> 1.4.18-2)
[2018-11-09 07:55] [ALPM] upgraded diffutils (3.6-1 -> 3.6-2)
[2018-11-09 07:55] [ALPM] upgraded gmp (6.1.2-1 -> 6.1.2-2)
[2018-11-09 07:55] [ALPM] upgraded autoconf (2.69-4 -> 2.69-5)
[2018-11-09 07:55] [ALPM] upgraded bison (3.1-1 -> 3.2-1)
[2018-11-09 07:55] [ALPM] upgraded gdbm (1.18-1 -> 1.18.1-1)
[2018-11-09 07:55] [ALPM] upgraded libffi (3.2.1-2 -> 3.2.1-3)
[2018-11-09 07:55] [ALPM] upgraded libutil-linux (2.32.1-2 -> 2.33-2)
[2018-11-09 07:55] [ALPM] upgraded python-requests (2.20.0-1 -> 2.20.1-1)
[2018-11-09 07:55] [ALPM] upgraded python-acme (0.27.1-1 -> 0.28.0-1)
[2018-11-09 07:55] [ALPM] upgraded findutils (4.6.0-2 -> 4.6.0-4)
[2018-11-09 07:55] [ALPM] upgraded libcap (2.25-1 -> 2.25-2)
[2018-11-09 07:55] [ALPM] upgraded certbot (0.27.1-1 -> 0.28.0-1)
[2018-11-09 07:55] [ALPM] upgraded cracklib (2.9.6-1 -> 2.9.6-3)
[2018-11-09 07:55] [ALPM] upgraded flex (2.6.4-1 -> 2.6.4-2)
[2018-11-09 07:55] [ALPM] installed libcroco (0.6.12+4+g9ad7287-1)
[2018-11-09 07:55] [ALPM] upgraded gettext (0.19.8.1-2 -> 0.19.8.1-3)
[2018-11-09 07:55] [ALPM] upgraded grep (3.1-1 -> 3.1-2)
[2018-11-09 07:55] [ALPM] upgraded groff (1.22.3-7 -> 1.22.3-8)
[2018-11-09 07:55] [ALPM] upgraded gzip (1.9-1 -> 1.9-2)
[2018-11-09 07:55] [ALPM] upgraded inetutils (1.9.4-5 -> 1.9.4-6)
[2018-11-09 07:55] [ALPM] upgraded kbd (2.0.4-1 -> 2.0.4-2)
[2018-11-09 07:55] [ALPM] upgraded libaio (0.3.111-1 -> 0.3.111-2)
[2018-11-09 07:55] [ALPM] upgraded libevent (2.1.8-1 -> 2.1.8-2)
[2018-11-09 07:55] [ALPM] upgraded lzo (2.10-1 -> 2.10-2)
[2018-11-09 07:55] [ALPM] upgraded libmariadbclient (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded libmnl (1.0.4-1 -> 1.0.4-2)
[2018-11-09 07:55] [ALPM] upgraded licenses (20171006-1 -> 20181104-1)
[2018-11-09 07:55] [ALPM] upgraded tar (1.30-1 -> 1.30-2)
[2018-11-09 07:55] [ALPM] upgraded make (4.2.1-2 -> 4.2.1-3)
[2018-11-09 07:55] [ALPM] upgraded mariadb-clients (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded mariadb (10.1.36-1 -> 10.1.37-1)
[2018-11-09 07:55] [ALPM] upgraded patch (2.7.6-3 -> 2.7.6-4)
[2018-11-09 07:55] [ALPM] upgraded util-linux (2.32.1-2 -> 2.33-2)
[2018-11-09 07:55] [ALPM] upgraded reiserfsprogs (3.6.27-1 -> 3.6.27-2)
[2018-11-09 07:55] [ALPM] upgraded sqlite (3.25.2-1 -> 3.25.3-1)
[2018-11-09 07:55] [ALPM] upgraded traceroute (2.1.0-1 -> 2.1.0-2)
[2018-11-09 07:55] [ALPM] upgraded zstd (1.3.6-1 -> 1.3.7-1)
iptables version:
iptables v1.8.0 (legacy)
audit records showing iptables-restore and ip6tables-restore are running during boot (although I don’t know why they’re generating 5 records each time; this also happens if I manually run the commands):
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd2b50 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd3240 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd3ad0 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cd4530 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[436]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=0 a2=40 a3=5565d0cdb840 items=0 ppid=1 pid=436 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty
=(none) ses=4294967295 comm="iptables-restor" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=iptables comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194ddf0 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tt
y=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194e360 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tt
y=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f194ff30 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f1950460 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)
Nov 16 14:17:42 archlinux audit[440]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=5 a1=29 a2=40 a3=55e8f1956340 items=0 ppid=1 pid=440 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip6tables-resto" exe="/usr/bin/xtables-legacy-multi" key=(null)