Arch Linux

broken

Member

Registered: 2018-09-10

Posts: 3

[Solved] Openssh-7.8p1-1 does not work but Openssh-7.7p1-(1/2)

Hi All,

I have some weird and not yet explained issue with the latest version of Openssh (7.8p1-1/2). After upgrading my system awhile ago, I am unable to connect to any host via SSH.

All connections results in the following error:
packet_write_wait: Connection to x.x.x.x port 22: Broken pipe

After some testing I decided to rollback to version 7.7p1-1 and all the connections work again, I have check the tickets against the latest version but this yield no results.

After digging through the forums I have seen the following two topics, which are similar but do not provide a fix:

- https://bbs.archlinux.org/viewtopic.php?id=240136
- https://bbs.archlinux.org/viewtopic.php?id=239982

When running SSH with the debug flags on I don't see anything obvious:

OpenSSH_7.8p1, OpenSSL 1.1.0i  14 Aug 2018
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 1: Applying options for *
debug1: /home/user/.ssh/config line 8: Applying options for SERVER
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "SERVER.test.domain.local" port 22
debug2: ssh_connect_direct
debug1: Connecting to SERVER.test.domain.local [x.x.x.x] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/SERVER_id_rsa type -1
debug1: identity file /home/user/.ssh/SERVER_id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.10
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.10 pat OpenSSH_5* compat 0x0c000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to SERVER.test.domain.local:22 as 'user'
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from SERVER.test.domain.local
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug3: send packet: type 30
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:aVBKSXvahDnhhVfzu7tuoAvP/Ombl/anDUh4HOTQ8ZM
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from SERVER.test.domain.local
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from x.x.x.x
debug1: Host 'SERVER.test.domain.local' is known and matches the ECDSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /home/user/.ssh/SERVER_id_rsa ((nil)), explicit
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/SERVER_id_rsa
Enter passphrase for key '/home/user/.ssh/SERVER_id_rsa': 
debug3: sign_and_send_pubkey: RSA SHA256:8Nw70JZ2xtzCY9loIWJVmjG/d4NgmkL+Dvz3pncAErM
debug3: sign_and_send_pubkey: signing using ssh-rsa
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to SERVER.test.domain.local ([x.x.x.x]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: exec
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug1: No xauth program.
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: send packet: type 1
packet_write_wait: Connection to x.x.x.x port 22: Broken pipe

While I can still work with the previous version, I would like to have this resolved in order to work with the latest version.

I am wondering if any of you have experienced the same behavior with the latest version of Openssh? And/or any suggestions of what I could to do debug this further?

//edit
The usual suspects of keep alive and interval has also been tested.

Last edited by broken (2018-11-19 07:05:26)

broken

Member

Registered: 2018-09-10

Posts: 3

Re: [Solved] Openssh-7.8p1-1 does not work but Openssh-7.7p1-(1/2)

Instant update, it seems that the current version of openssh does not work with RSA keys.
Other key types work file, time to migrate to ed25519.

I made a horrible mistake while testing, moving back and forward between version for testing caused the false positive. A coworker was able to simulate the issue with an RSA key on his machine, testing it on the other machines resulted in the failure posted in the first post. Hence the confusion.

Last edited by broken (2018-09-10 12:52:18)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 21,616

Re: [Solved] Openssh-7.8p1-1 does not work but Openssh-7.7p1-(1/2)

Moving to Networking

broken

Member

Registered: 2018-09-10

Posts: 3

Re: [Solved] Openssh-7.8p1-1 does not work but Openssh-7.7p1-(1/2)

The issue has been resolved. The following in my ssh_config file allows me to work with the newer versions of openssh:

Host *
    IPQoS throughput

This seems to conflict between version with my setup.