[SOLVED] Unable to connect to local http server hosted on Arch

Anon_a · 2024-07-23 04:47:05

I have recently started messing with firewalls and I thought that this could be the main cause, but after disabling everything the problem persists.

I installed iptables-nft and firwalld, after which running:

 firewall-cmd --add--port=8000/tcp

was usually enough to get it to work and connect from other local computers on the network, but since recently nothing I do seems to allow incoming connections so I'm not sure if it could be due to an update. There are no error messages when running:

systemctl status firewalld
systemctl status iptables
systemctl status nftables

and after running:

systemctl stop firewalld
iptables -F

I'm still unable to connect to the Arch machine. I don't even know where to start looking for the problem as nothing is shown in the logs, so any help would be appreciated.

Last edited by Anon_a (2024-07-23 19:26:18)

seth · 2024-07-23 06:22:07

On the host supposedly running the http server

ss -pentl # Is anything listening at all?
curl -v http://localhost:8000 > /dev/null # what does the http server respond

From the other host

nmap <server ip>

There are no error messages when running:

nftables ships a default rule that iir only allows sshd and firewalld will ignore that as will iptables -F
https://wiki.archlinux.org/title/Nftabl … e_firewall
Running several filter services/tools greatly complicates things. Configure your firewall through a single vector.
https://wiki.archlinux.org/title/Nftables#Usage

Anon_a · 2024-07-23 07:41:27

I did test the local server using a custom server and using python:

python -m http.server

and was able to connect on the Arch machine itself using a browser using both methods, but not from another machine on the same network.

seth wrote:

From the other host
nmap <server ip>

I'm assuming you mean that this should be run from the machine I'm connecting from? I am not connecting from another Linux machine and I don't really have one available so I can't really do this.

Is there any way I can see which incoming connections are being made or rejected from the host machine? Or at least see what might be causing the problem?

Last edited by Anon_a (2024-07-23 07:48:08)

seth · 2024-07-23 07:49:15

I am not connecting from another Linux machine and I don't really have one available so I can't really do this.

And what are you connecting from? (nmap exists for windows, macos and adroid and I guess iOS?)
Either way, if you've a service responding on the local port, check the nftables situation.

Edit:

Is there any way I can see which incoming connections are being made or rejected from the host machine?

wireshark, but while you're unspecific about "no errors w/ the nftables.service" I'm betting your right arm that you've loaded the default nftables config which blocks pretty much everything but ssh

Last edited by seth (2024-07-23 07:50:32)

Anon_a · 2024-07-23 09:16:48

seth wrote:

I'm betting your right arm that you've loaded the default nftables config which blocks pretty much everything but ssh

I am running nftables with firewalld so this is likely, but how would I check to see if all the rules are cleared? From what I've read running iptables -F should clear the rules, and I also tried commenting out and reloading the config in

/etc/nftables.conf

seth wrote:

And what are you connecting from? (nmap exists for windows, macos and adroid and I guess iOS?)

I'm connecting from iOS, but I doubt it. I'll see if I can use wireshark to track down the problem, but I think it's more likely that there is some rule that I'm not clearing properly as you suggested. I think I also tried

# nft flush ruleset

, but I'll double check this later and look at the output from

# nft list ruleset

I should probably also mention that I tried adding the http and https services to the ruleset using:

sudo firewall-cmd --add-service=http
sudo firewall-cmd --add-service=https

but this was not required previously for the server to be reachable on the network.

Last edited by Anon_a (2024-07-23 09:20:19)

Whoracle · 2024-07-23 10:18:12

Have a look at the above mentioned

ss -pentl

or

netstat -tlpen

to see how exactly your python webserver listens. Maybe it's only listening on localhost, or for some reason only on IPv6 or something. You need to see it listening on 0.0.0.0 or a specific interface IP.

I know it's basics, but I have personally overlooked stupider things, so...

cryptearth · 2024-07-23 10:23:53

I can be wrong - but on my servers (opensuse) I have to use --persistent option along with a reload after any modifications
without that it only affects the current instance and is reset on reload

seth · 2024-07-23 14:17:44

I am running nftables with firewalld so this is likely, but how would I check to see if all the rules are cleared? From what I've read running iptables -F should

iptables does jacksquat about most nf tables, firewalld implements and exlcusively controls its own and all of the magic actually happens in the kernels netfilter - everything userspace just controls those and using mutliple tools for that tends to be a pita.

Disable the nftables.service (it has nothing to do with firewalld needs it to function or stuff like that) and reboot.
"nft flush ruleset" would flush all rules but also see https://wiki.archlinux.org/title/Nftables#Tables

@Whoracle, the OP claims to have access to the service on the localhost, so something's listening there but there's no external access
@cryptearth, the IP filters are transient by nature. They need to be re-loaded from userspace w/ every reboot and the various abstraction levels (iptables, nft/nftables and the fancy firewalld/ufw etc) will do exactly to (next to making it a bit easier to manage those rules)

Anon_a · 2024-07-23 17:20:39

seth wrote:

"nft flush ruleset" would flush all rules

So flushing the nftables ruleset worked and I was able to connect to the local http server. So the problem seems to be with firewalld, but it doesn't really explain why adding the 8000 tcp port stopped working?

I flushed the nftables ruleset, restarted firewalld and added the 8000 tcp port, here's the relevant extract from "sudo nft list ruleset":

chain filter_INPUT {
    type filter hook input priority filter + 10; policy accept;
    ct state { established, related } accept
    ct status dnat accept
    iifname "lo" accept
    ct state invalid drop
    jump filter_INPUT_POLICIES
    reject with icmpx admin-prohibited
}

...

chain filter_INPUT_POLICIES {
    iifname "xxxxxxx" jump filter_IN_policy_allow-host-ipv6
    iifname "xxxxxxx" jump filter_IN_public
    iifname "xxxxxxx" reject with icmpx admin-prohibited
    jump filter_IN_policy_allow-host-ipv6
    jump filter_IN_public
    reject with icmpx admin-prohibited
}

...

chain filter_IN_public {
    jump filter_IN_public_pre
    jump filter_IN_public_log
    jump filter_IN_public_deny
    jump filter_IN_public_allow
    jump filter_IN_public_post
    meta l4proto { icmp, ipv6-icmp } accept
}

chain filter_IN_public_pre {
}

chain filter_IN_public_log {
}

chain filter_IN_public_deny {
}

chain filter_IN_public_allow {
    tcp dport 22 accept
    ip6 daddr fe80::/64 udp dport 546 accept
    tcp dport 8000 accept
}

chain filter_IN_public_post {
}

...

This looks exactly like it did when this approach worked so I don't know what changed or what I'm doing wrong?

Last edited by Anon_a (2024-07-23 17:24:05)

Whoracle · 2024-07-23 17:41:54

seth wrote:

@Whoracle, the OP claims to have access to the service on the localhost, so something's listening there but there's no external access

Yes, on the localhost. but how has so far not been shown. Do they use http://localhost:8000? http:://IP:8000? http://hostname:8000 that /ect/hosts or something maps to ::1? Maybe a /etc/hosts entry from some earlier debugging or project that they forgot about? ss or netstat would at least rule those out instead of assuming.

Especially since they can't access the app even after disabling the firewall this seems more and more likely.

Another thought: AppArmor or SELinux maybe? Years and Years ago I had issues with CentOS that came with overzealous default profiles for one or the other, so maybe that was installed and somehow borked?

Anon_a · 2024-07-23 18:06:11

Whoracle wrote:

Yes, on the localhost. but how has so far not been shown. Do they use http://localhost:8000? http:://IP:8000? http://hostname:8000 that /ect/hosts or something maps to ::1? Maybe a /etc/hosts entry from some earlier debugging or project that they forgot about? ss or netstat would at least rule those out instead of assuming.

I used the http://IP:8000 version on the hosting machine, but the problem seems to be with nftables or firewalld, see my previous reply. The problem is not with the server itself, but with the firewall it seems.

Whoracle · 2024-07-23 18:17:35

Since I positively hate local machine firewalls, I'll not be of much help there, so I'll offer some suggestions "outside the box" so to speak, and let others help out with the firewall itself, then. Did you install AppArmor or SELinux or any other security software other than the firewall?

Also, maybe go back to basics, backup your nftables config somewhere and just comletely uninstall it, just to make 100% sure it's that? Because if I understood your opening post correctly, even disabling nftables doesn't help, so something's got to be lingering.

Anon_a · 2024-07-23 18:24:14

Whoracle wrote:

Since I positively hate local machine firewalls, I'll not be of much help there, so I'll offer some suggestions "outside the box" so to speak, and let others help out with the firewall itself, then. Did you install AppArmor or SELinux or any other security software other than the firewall?
Also, maybe go back to basics, backup your nftables config somewhere and just comletely uninstall it, just to make 100% sure it's that? Because if I understood your opening post correctly, even disabling nftables doesn't help, so something's got to be lingering.

So it seems like simply stopping nftables does not flush the rules, but running

# nft flush ruleset

does clear it and I'm then able to connect on the same network. I did a fresh install about two weeks ago. I did not use SELinux or set up AppArmor on this install. The only security based packages I installed was "firewalld" and "iptables-nft". Simply adding the 8000 tcp port to firewalld was enough to be able to connect within the network for the past two weeks after installing, but since today it stopped working for some reason so something must have changed.

Whoracle · 2024-07-23 18:33:55

OK, so it is nftables then. One thing I seem to vaguely remember from the issues we had at work with firewalld/nftables lately is that when binding to only one IP instead of 0.0.0.0 we needed to set a zone somewhere. Your config allows port 22 incoming, but I guess sshd is listening on 0.0.0.0, and your app is only bound to SOME_SPECIFIC_IP?

Also, just to cover all the bases: your python httpd is running natively, not via docker or something? And the httpd does not have something like "allowed origins" or the like configured? What happens if you, as a test, install nginx, rebind it to port 8000 and try to access it? You should get the satatic default nginx website.

Anon_a · 2024-07-23 18:54:33

Whoracle wrote:

OK, so it is nftables then. One thing I seem to vaguely remember from the issues we had at work with firewalld/nftables lately is that when binding to only one IP instead of 0.0.0.0 we needed to set a zone somewhere. Your config allows port 22 incoming, but I guess sshd is listening on 0.0.0.0, and your app is only bound to SOME_SPECIFIC_IP?
Also, just to cover all the bases: your python httpd is running natively, not via docker or something? And the httpd does not have something like "allowed origins" or the like configured? What happens if you, as a test, install nginx, rebind it to port 8000 and try to access it? You should get the satatic default nginx website.

I am not using docker and I'm currently using

python -m http.server

to test, which from the documentation binds to all interfaces and reports that it's serving on:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000) ...

and from the config output from nft it seems like it added the port alongside the ssh one, so it should work, but unfortunately I don't have another computer available to test if ssh works.

I also tried adding the 8000 tcp port to all zones except for "drop" and "block", but it still doesn't connect. I also made sure that sshd is not running and I also removed the ssh service from all zones and confirmed that port 22 is no longer accepted in the nft rules, only the 8000 tcp port remained.

Anon_a · 2024-07-23 19:14:46

So after some further testing the issue seems to be with firewalld.

I added the following to the default nftables config file in /etc/nftables.conf:

table inet filter {
...
    tcp dport 8000 accept
...
}

After which stopping firewalld and restarting nftables seems to allow connections on the tcp 8000 port. So I might not be adding the port correctly to firewalld or something in the way it adds the rules to nftables changed.

Anon_a · 2024-07-23 19:24:56

So I think I solved it. The main problem seems to have been that firewalld appends it's config after reading in the config in /etc/nftables.conf, which even after I commented out the config and reloading all the services did not clear the actual rules being used, which I saw after looking at the first few lines produced by

# nft list ruleset

So after commenting out the rules in /etc/nftables.conf, flushing the nftables ruleset with "nft flush ruleset", reloading both nftables and firewalld and after adding the tcp 8000 port with firewalld I can finally connect.

So at some point in the past few days either the config in /etc/nftables.conf reset and since it does not seem to clear the rules after stopping the service the rules persisted after everything I tried, which was why nothing seemed to have worked.

Thank you everyone for pointing me in the right direction.

Last edited by Anon_a (2024-07-23 19:32:56)

seth · 2024-07-23 20:42:58

it's config after reading in the config in /etc/nftables.conf

That file is loaded by nftables.service
Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

(nb. I didn't read most of the posts between here and your last one)

Anon_a · 2024-07-24 04:19:01

seth wrote:

Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Here is the output:

auditd.service                           | multi-user.target.wants
dbus-org.fedoraproject.FirewallD1.service | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.timesync1.service   | system
firewalld.service                        | multi-user.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
iptables.service                         | multi-user.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
nftables.service                         | multi-user.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
remote-fs.target                         | multi-user.target.wants
systemd-timesyncd.service                | sysinit.target.wants
systemd-userdbd.socket                   | sockets.target.wants
wireplumber.service                      | pipewire.service.wants
xdg-user-dirs-update.service             | default.target.wants

I'm assuming the nftables service should not be enabled?

seth · 2024-07-24 05:46:51

I'm assuming the nftables service should not be enabled?

Correct.
Neither should be iptables.service

Arch Linux

#1 2024-07-23 04:47:05

[SOLVED] Unable to connect to local http server hosted on Arch

#2 2024-07-23 06:22:07

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#3 2024-07-23 07:41:27

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#4 2024-07-23 07:49:15

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#5 2024-07-23 09:16:48

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#6 2024-07-23 10:18:12

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#7 2024-07-23 10:23:53

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#8 2024-07-23 14:17:44

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#9 2024-07-23 17:20:39

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#10 2024-07-23 17:41:54

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#11 2024-07-23 18:06:11

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#12 2024-07-23 18:17:35

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#13 2024-07-23 18:24:14

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#14 2024-07-23 18:33:55

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#15 2024-07-23 18:54:33

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#16 2024-07-23 19:14:46

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#17 2024-07-23 19:24:56

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#18 2024-07-23 20:42:58

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#19 2024-07-24 04:19:01

Re: [SOLVED] Unable to connect to local http server hosted on Arch

#20 2024-07-24 05:46:51

Re: [SOLVED] Unable to connect to local http server hosted on Arch

Board footer