[Resolved] Network Connection Unstable/Intermediate

eduncan911 · 2015-09-28 19:57:54

Got an odd problem. System was working perfectly fine. I could ssh using a private key from my laptop all day long, reboot, shutdown, startup, etc. Zero issues with ssh for about two weeks.

Then I:

* installing lm_sensors
* installed qemu-minimal from AUR
* configured kernel modules for virtio (part of qemu)
* setup a bridge on 2nd interface

Rebooted. And I can no longer ssh to the box reliably...

$ ssh server
ssh: connect to host server port 22: Connection refused

$ ssh server
Connection reset by 192.168.1.10

...but if I keep at it, over and over again:

$ ssh server
Last login: Mon Sep 28 15:43:03 2015 from 192.168.1.166
[eric@server ~] $

But, I get an error within a few minutes:

packet_write_wait: Connection to 192.168.1.10: Broken pipe

If I ping the server from laptop, I get 0% lost - for hours. Every ping succeeds. Even while I get the connection reset and refused errors, I can ping 100% of the time.

- "journalctl -f" shows zero/no activity from sshd when I cannot connect. but when I do connect, it shows the log of that session.
- I can ping the server remotely, I can log into the server and ping google.com, I can telnet locally to port 22.
- "ps aux | grep sshd" shows it is running on the remote server (but I am logged in at this time, so ssh actually works)
- removing the bridge I setup previously and going back to "what worked before" has no affect after reboot - same problem.
- removed qemu-minimal package, and all dependencies, rebooted, no effect.
- removed kvm and virtio kernel models, rebuilt mkinitcpio and rebooted, no effect.
- using systemd-networkd for networking.

The only thing I can think of is the qemu-minimal Post-Install script that returned an error when I installed it. Perhaps it was configuring something in the system that I haven't been able to track down yet (netctl?).

https://aur.archlinux.org/packages/qemu-minimal/
^- see my comment about the error there.

I extracted the packaged and looked around; but, i was not familiar enough to know what is going on with the files. I see it did install some kind of network driver.

Thanks in advance!

EDIT: Must be an sshd issue. If I logout of the tty1 console, about half the time I get disconnected from a live ssh connection I have already established.

Here's one with a "connection refused" responses:

$ ssh -vvv server
OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /home/eric/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to server [192.168.1.10] port 22.
debug1: connect to address 192.168.1.10 port 22: Connection refused
debug1: Connecting to server [192.168.1.10] port 22.
debug1: connect to address 192.168.1.10 port 22: Connection refused
ssh: connect to host server port 22: Connection refused

Ok, it simply can't connect with that one. But, PING constantly returns a response?!?

Here's a "connection reset" response:

$ ssh -vvv server
OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /home/eric/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to server [192.168.1.10] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.1
debug1: match: OpenSSH_7.1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to server:22 as 'eric'
debug3: hostkeys_foreach: reading file "/home/eric/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/eric/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from server
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
Connection reset by 192.168.1.10

Ok, that looks like it connected but got disconnected in the middle of sending a data.

And a "connection reset by peer" response:

$ ssh -vvv server
OpenSSH_7.1p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /home/eric/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to server [192.168.1.10] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/eric/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
ssh_exchange_identification: read: Connection reset by peer

^- all three were taken within about 30 seconds of multiple attempts at sshing. Please note that PING constantly returns a response and journalctl shows no activity at all being logged.

Wait 30 seconds, and I can connect. Wait 30 seconds, and I am disconnected from an established connection.

journalctl shows absolutely zero messages from any service anywhere. If I "sudo" something from console, then it updates.

Is sshd logging to somewhere else?

Last edited by eduncan911 (2015-09-29 03:48:26)

nixpunk · 2015-09-28 20:44:56

Please post the output of an ssh connection attempt with verbosity turned on to the max (-vvv). Also, are you using any kind of encryption, or other mechanisms that require authentication when logging in locally? How about weirdness coming from .bash_profile or any other file sourced at login?

eduncan911 · 2015-09-28 20:59:58

Updated OP with ssh -vvv (I was in the middle of doing just that! :) ).

No encryption. Just a bare Arch install, 1 user (me).

No funny business in .bash_profile (it's bare).

Again, this was all working 100% (for a week, several reboots, tinkering with config files, etc) before I installed qemu-minimal + setting up the bridge on a 2nd ethernet. I've reverted the bridge and ethernet changes; but, the problem persists.

I am heavily leaning towards some network driver that qemu-minimal installed. But, I am not sure how to debug that.

sshd_config is very minimal:

$ cat /etc/ssh/sshd_config 
#	$OpenBSD: sshd_config,v 1.97 2015/08/06 14:53:21 deraadt Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# The default requires explicit activation of protocol 1
#Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Ciphers and keying
#RekeyLimit default none

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#RSAAuthentication yes
#PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
PermitEmptyPasswords no

# Change to no to disable s/key passwords
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no # pam does that
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
UsePrivilegeSeparation sandbox		# Default for new installations.
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	/usr/lib/ssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

Also note that the ssh connection that I can get established, after logging into the console, is not stable. It disconnects at random times and I cannot re-dial/ssh back into the server for some time. And then, it just "happens" to start working again.

EDIT: If I reboot and just 'wait' for 3 minutes, i can ssh in. Then it disconnects, and I cannot.

Network configuration:

[eric@server ~] $ ls -l /etc/systemd/network/
total 4
-rw-r--r-- 1 root root 87 Sep 27 23:14 eno1.network

[eric@server ~] $ cat /etc/systemd/network/eno1.network 
[Match]
Name=eno1

[Network]
Address=192.168.1.10/24
Gateway=192.168.1.1
DNS=192.168.1.1

It can't get any easier than that!

Last edited by eduncan911 (2015-09-28 21:07:28)

eduncan911 · 2015-09-29 00:59:15

Resolved... An old VM i had running on an old server was using this IP address. *shakes head*

Now to get back the last 24 hours of grief...

Thank you all for looking.

Last edited by eduncan911 (2015-09-29 03:48:02)

Arch Linux

#1 2015-09-28 19:57:54

[Resolved] Network Connection Unstable/Intermediate

#2 2015-09-28 20:44:56

Re: [Resolved] Network Connection Unstable/Intermediate

#3 2015-09-28 20:59:58

Re: [Resolved] Network Connection Unstable/Intermediate

#4 2015-09-29 00:59:15

Re: [Resolved] Network Connection Unstable/Intermediate

Board footer