[solved] sshd problem - any help greatly appreciated

4rty · 2021-04-16 05:25:43

Solved - user error - i fucked up and sorry for wasting everyone's time.

Hi everybody,
So I have a pretty fresh install of Arch (maybe a week or 2 old now) and I had ssh server up and running no problem until the PC I was logged in from crashed and now I cant log back in, just a standard "connection refused" message. Edit: using putty I get standard connection refused, using powershell i get connection refused, ant using terminus (which I usually use I used putty and powershell to troubleshoot maybe it was the client I thought at one point) i get connect ECONNREFUSED 192.168.1.162:22 --> Remote rejected opening a shell channel: Error: Not Connected
I did all the standard troubleshooting like making sure the service is up and running, port open (verified with nmap), ip conflicts check (arp-scan), the server and client on same LAN I don't open these machines/ports to WAN, I also ssh'd into the server from itself to make sure the install and config was good. So I checked everything I could think of but still nothing.
I then checked with 'journalctl -u sshd' and got these messages:

pam_unix(sudo:session): session opened for user root(uid=0) by art(uid=1000)
pam_unix(sudo:session): session closed by user root
pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
art : TTY=pts/0 ; PWD=/srv/http ; USER=root ; COMMAND=/usr/bin/systemctl status sshd
pam_unix(sudo:session): session opened for user root(uid=0) by art(uid=1000)
pam_unix(sudo:session): session closed by user root
pam_systemd_home(sudo:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
art : TTY=pts/0 ; PWD=/srv/http ; USER=root ; COMMAND=/usr/bin/iptables --list-rules
Accepted password for art from 192.168.1.162 port 56216 ssh2
Received disconnect from 192.168.1.162 port 56216:11: disconnected by user
Disconnected from user art 192.168.1.162 port 56216
Stopping OpenSSH Daemon ...
sshd.service: Deactivated successfully.
Stopped OpenSSH d-Daemon.
sshd.service: Consumed 11.869s CPU time.
-- Boot bunchanumbers --
Started OpenSSH Daemon
Server listening on 0.0.0.0 port 22
Accepted password for art from 192.168.1.162 port 45030 ssh2
pam_unix(sudo:auth): conversation failed
pam_unix(sudo:auth): auth could not identify password for [art]
pam_systemd_home(sudo:auth): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Received disconnect from 192.168.1.162 port 45030:11: disconnected by user
Disconnected from user art 192.168.1.162 port 45030

Edit: By the way the Arch machine has static IP of 192.168.1.162 .... in case that is any help

So I searched around for pam, but I never activated pam and in fact I specifically remember changing the sshd_config file to specifically tell it that I don't want to use pam right now.

Some additional info that might help: I did generate public private keys using ssh-keygen ed25519 in order to migrate over to using pam and key/signature verification in he future but I never changed any configs to reflect that and havent restarted the machine/daemon since I generated those keys (except when I rebooted but the issue started before that reboot).

Any help would be greatly appreciated.

Last edited by 4rty (2021-04-17 16:45:41)

seth · 2021-04-16 08:24:25

For clarification: the problem is limited to the one client that "crashed" during an ssh session?
Can you connect from other clients in the LAN?

4rty · 2021-04-16 08:36:50

I'm sorry no, tried through various other methods as well.
Just to be exhaustive and provide as much info as possible; other methods tried were:
client refrenced in original post was windows 10 using terminus, putty, powershell
android using juicyssh and terminus while 1) connected directly to LAN via wifi and 2) mobile network to local VPN (mobile network to LAN via VPN [OpenVPN])
ssh into raspberry pi and then into arch machine (rasbian; everything, meaning kernel and all packages kept up to date pretty regularly)
all failed, can get exact failure/erro responses if needed
So essentially I can ssh into itself only if on the machine itself physically ... if that makes sense .... like its setup on an old laptop that I used as an arch linux server in the past, so when i open it up and physically use the keyboard to ssh into itself it works and then exit out of tunnel using 'exit'

Last edited by 4rty (2021-04-16 08:55:30)

seth · 2021-04-16 08:47:28

I just wanted to sort ot whether this is client or a server issue.
Can you log into a different account (basically "useradd -m art2")?

4rty · 2021-04-16 08:59:38

Just added new user art2 set password (no group or sudo) and same error- connection refused - again tried on multiple applications and machines using new user (no sudo)

ran 'journalctl -u sshd' nothing new from what i posted above (originaly) - figured I might get more info from this login attempt?

Last edited by 4rty (2021-04-16 09:04:29)

seth · 2021-04-16 09:21:22

https://wiki.archlinux.org/index.php/OpenSSH#Checklist point #5
If the new user doesn't show up in the logs, use a different filter: https://wiki.archlinux.org/index.php/Op … management (absent timestamps, that could be old-ish stuff)

4rty · 2021-04-16 09:43:42

OK, so,
user art2 did not have a .ssh folder so i created one using root gave it the correct permissions and made art2 the owner, also changed loglevel to debug in sshd config file,
but nothing changes in logs when using 'journal -u sshd' - i restarted the sshd daemon and that shows up but nothing else
as far as 'journactl -xe' i get

sshd[837]: debug1: Bind to port 22 on 0.0.0.0.
systemd[1]: sshd.service: Deactivated successfully
some green text that doesn't appear t show any additional usefull info, appears to be verbose version of previous 2 commands - but I can copy here if neccessary
sshd[837]: Server listening on 0.0.0.0 port 22
systemd[1]: Stopped OpenSSH Daemon.
again green text just appears to be verbose version of previous 2 entries
sshd[837]: debug1: Bind to port 22 on ::.
systemd[1]: Condition check resulted in SSH Key Generation being skipped.
again verbose green text
sshd[837]: Server listening on :: port 22.
systemd[1] Started OpenSSH Daemon.
again green verbose version

so still can't connect using art or art2 ... makes me thing its getting blocked somehow because logs dont show login attempts? But i reverified that port 22 is open using nmap so i don't know

Last edited by 4rty (2021-04-16 12:01:52)

4rty · 2021-04-16 12:00:47

Bump, sorry if this is against the rules - I edited the previous post that was kind of a place holder for my reply when I left to take care of something...

so still at the point where i can ssh using either users from the machine itself but not elsewhere - doesnt make sense to me ...

I have a lot of debug lines now from when i ssh'ed from the machine itself but no logs from anything else ...

journalctl -xe output : http://ix.io/2Wi2 the connection on the bottom is from ssh from the machine itself not from an external connection

Last edited by 4rty (2021-04-16 12:44:23)

4rty · 2021-04-16 13:34:57

tcpdump -ni enp4s0 "port 22" - is capturing packets so it's not being blocked by anything ..... I am truly at a loss here
Also i tried to save the tcpdump into a file but i clearly did something wrong because it messed up my shell and any input from keyboard was mapped wrong ... I assume i saved a binary file and when i cat out it messed with it? I dont know but I had to reboot

Last edited by 4rty (2021-04-16 14:48:38)

seth · 2021-04-16 15:41:46

Run

sudo journalctl -f

and try to ssh in, you should™ see the server logging that?
And please post your sshd_config (please mark lines that you obfuscated for privacy reasons)

4rty · 2021-04-16 19:19:10

sshd_config : http://ix.io/2WjY and no lines obfuscated as no private data was in there ... should there have been?
no luck with journalctl -f showing login attempts
I'm using root for debugging as I believe this is one of the scenario's it's warranted so no need for sudo
I want to thank you for your help so far

EDIT: 'journalctl -f' output : http://ix.io/2Wlv

Last edited by 4rty (2021-04-17 00:52:35)

seth · 2021-04-17 07:21:36

Since it's not the clients, not the logins, you can ssh locally, there's no IP restriction in the sshd config, you receive packages on post 22 and ssh doesn't log that interaction even in debug mode…

iptables -nvL
nft list ruleset

Local firewall messed up?

tucuxi · 2021-04-17 08:55:09

IP address conflict?

4rty · 2021-04-17 16:06:37

so seth I think you found the culprit, I'm not very good at iptables, but a quick look at it makes me think that's the problem because I don't see a rule for port 22, although there is 1 TCP refrence and I believe that is a refrence to port 22.
I followed this tutorial setting it up: https://wiki.archlinux.org/index.php/Si … l_firewall - I did drop all inbound deny all outbound accept and then inbound port 22 TCP accept as per the tutorial and it worked fine for a week or so I'm not sure what I could have done to mess it up though

bash doesn't recognize command 'nft' when running

nft list ruleset

output of iptables -nvL

Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
55 4266 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT 41 -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
3 180 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8 ctstate NEW
30356 2402K UDP udp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW
20 1040 TCP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 ctstate NEW
30356 2402K REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
20 1040 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with tcp-reset
2369 85284 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-proto-unreachable
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 16194 packets, 1720K bytes)
pkts bytes target prot opt in out source destination
Chain TCP (1 references)
pkts bytes target prot opt in out source destination
Chain UDP (1 references)
pkts bytes target prot opt in out source destination

also tcpdump showed that I was receiving packets on port 22 when I tried to login, if it was being blocked because of firewall would it not have shown anything?

and tucuxi I tried to rule that out running arp-scan from another machine and it didn't find anything else with that ip

Last edited by 4rty (2021-04-17 16:15:10)

seth · 2021-04-17 16:18:00

Looking at the wiki and you list, you probably blindly entered all the babyblue boxes - but they save the rules into a file (which you're going to re-load on boots) *before* they add

-A TCP -p tcp --dport 22 -j ACCEPT

so the moment you reboot and load the iptables from the rules, inbound on port 22 gets dropped.

You can still see those packets incoming (because they do) but then your firewall tells them that they're going nowhere from here on.

Edit: the entire wiki is educational. There're examples but you can't use them as a blueprint.

Last edited by seth (2021-04-17 16:19:15)

4rty · 2021-04-17 16:26:58

Your right I did use them as blueprint but I did carefully read through what I was entering to make sure I knew what I was entering so my question now is why do all rules except

-A TCP -p tcp --dport 22 -j ACCEPT

get loaded? I'm pretty sure I ran iptables-save before rebooting or I rebooted after saving the iptables rules to verify that they would load. Also I never rebooted between ssh sessions so that raises another question.

Last edited by 4rty (2021-04-17 16:37:23)

4rty · 2021-04-17 16:33:15

seth was right, I was pretty confident I saved the last rule but apparently not, this is embarrassing and sorry for wasting everyone's time.

.bash_history tells me ....
rather than running

iptables-save -f /etc/iptables/iptables.rules

I ran only

iptables-save

cause I'm a noob sorry

Last edited by 4rty (2021-04-17 16:50:14)

Arch Linux

#1 2021-04-16 05:25:43

[solved] sshd problem - any help greatly appreciated

#2 2021-04-16 08:24:25

Re: [solved] sshd problem - any help greatly appreciated

#3 2021-04-16 08:36:50

Re: [solved] sshd problem - any help greatly appreciated

#4 2021-04-16 08:47:28

Re: [solved] sshd problem - any help greatly appreciated

#5 2021-04-16 08:59:38

Re: [solved] sshd problem - any help greatly appreciated

#6 2021-04-16 09:21:22

Re: [solved] sshd problem - any help greatly appreciated

#7 2021-04-16 09:43:42

Re: [solved] sshd problem - any help greatly appreciated

#8 2021-04-16 12:00:47

Re: [solved] sshd problem - any help greatly appreciated

#9 2021-04-16 13:34:57

Re: [solved] sshd problem - any help greatly appreciated

#10 2021-04-16 15:41:46

Re: [solved] sshd problem - any help greatly appreciated

#11 2021-04-16 19:19:10

Re: [solved] sshd problem - any help greatly appreciated

#12 2021-04-17 07:21:36

Re: [solved] sshd problem - any help greatly appreciated

#13 2021-04-17 08:55:09

Re: [solved] sshd problem - any help greatly appreciated

#14 2021-04-17 16:06:37

Re: [solved] sshd problem - any help greatly appreciated

#15 2021-04-17 16:18:00

Re: [solved] sshd problem - any help greatly appreciated

#16 2021-04-17 16:26:58

Re: [solved] sshd problem - any help greatly appreciated

#17 2021-04-17 16:33:15

Re: [solved] sshd problem - any help greatly appreciated

Board footer