[SOLVED] SSH problem with archiso with sshd enabled for system mainten

curtisleebolin · 2014-09-07 02:45:36

I have created an archiso with sshd enable and a default root password set for maintenance on some headless servers, but SSH does not accept the password.
(Please don't hound me about security, this is only booted when the server's nic is directly connected to a maintenance laptop.)

The airootfs/root/customize_airootfs.sh for enabling sshd and setting the default root password.

#!/bin/bash

set -e -u

sed -i 's/#\(en_US\.UTF-8\)/\1/' /etc/locale.gen
locale-gen

ln -sf /usr/share/zoneinfo/US/Central /etc/localtime

usermod -s /usr/bin/bash root
cp -aT /etc/skel/ /root/
chmod 700 /root
echo -en 'root\nroot' | passwd

sed -i "s/#Server/Server/g" /etc/pacman.d/mirrorlist
sed -i 's/#\(Storage=\)auto/\1volatile/' /etc/systemd/journald.conf

systemctl enable pacman-init.service choose-mirror.service sshd.service
systemctl set-default multi-user.target

Error on client side (testing in VM):

$ ssh -vvv root@192.168.122.246
OpenSSH_6.6.1, OpenSSL 1.0.1i 6 Aug 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.122.246 [192.168.122.246] port 22.
debug1: Connection established.
debug1: identity file /home/lee/.ssh/id_rsa type -1
debug1: identity file /home/lee/.ssh/id_rsa-cert type -1
debug1: identity file /home/lee/.ssh/id_dsa type -1
debug1: identity file /home/lee/.ssh/id_dsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/lee/.ssh/id_ecdsa" as a RSA1 public key
debug1: identity file /home/lee/.ssh/id_ecdsa type 3
debug1: identity file /home/lee/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/lee/.ssh/id_ed25519 type -1
debug1: identity file /home/lee/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.168.122.246" from file "/home/lee/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/lee/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@o
penssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-
128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-
128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 52:eb:3e:0a:04:fe:8e:9d:ba:1d:51:c7:da:2e:b1:37
debug3: load_hostkeys: loading entries for host "192.168.122.246" from file "/home/lee/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/lee/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host '192.168.122.246' is known and matches the ECDSA host key.
debug1: Found key in /home/lee/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/lee/.ssh/id_rsa ((nil)),
debug2: key: /home/lee/.ssh/id_dsa ((nil)),
debug2: key: /home/lee/.ssh/id_ecdsa (0x7fb294c90c80),
debug2: key: /home/lee/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/lee/.ssh/id_rsa
debug3: no such identity: /home/lee/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/lee/.ssh/id_dsa
debug3: no such identity: /home/lee/.ssh/id_dsa: No such file or directory
debug1: Offering ECDSA public key: /home/lee/.ssh/id_ecdsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/lee/.ssh/id_ed25519
debug3: no such identity: /home/lee/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@192.168.122.246's password: 
debug3: packet_send2: adding 64 (len 49 padlen 15 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
root@192.168.122.246's password:

Error on server side:

# journalctl /usr/bin/sshd
Sep 06 21:18:25 archiso sshd[426]: Failed password for root from 192.168.122.1 port 55274 ssh2

I also ran sshd with debug:

debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1i 6 Aug 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/bin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.122.1 port 55364 on 192.168.122.246 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: permanently_set_uid: 99/99 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none [preauth]
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "192.168.122.1"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: test whether pkalg/pkblob are acceptable [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys '/root/.ssh/authorized_keys': No such file or directory
debug1: restore_uid: 0/0
Failed publickey for root from 192.168.122.1 port 55364 ssh2: ECDSA 57:6c:e3:a6:87:1e:26:f1:fa:10:bf:cc:57:73:f5:23
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 2 failures 1 [preauth]
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 192.168.122.1 port 55364 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 3 failures 2 [preauth]
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 192.168.122.1 port 55364 ssh2
debug1: userauth-request for user root service ssh-connection method password [preauth]
debug1: attempt 4 failures 3 [preauth]
debug1: PAM: password authentication failed for root: Authentication failure
Failed password for root from 192.168.122.1 port 55364 ssh2
Connection closed by 192.168.122.1 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 520

I can see the hash in /etc/shadow so I tested the password and it was accepted:

[root@archiso ~]# useradd -s /usr/bin/bash foo
[root@archiso ~]# su foo
[foo@archiso ~]$ su
Password:
[root@archiso ~]#

I have no idea what the issue is. Please try to point me in the right direction if you have an idea.

Thank You For Your Time.

Last edited by curtisleebolin (2014-09-07 03:41:57)

curtisleebolin · 2014-09-07 03:17:02

If I add UsePAM no to /etc/ssh/sshd_config it works fine.

I don't want to put a static sshd_config file on my iso, since I generate a new iso each time I need to do maintenance and want the latest sshd_config from the openssl package.

fukawi2 · 2014-09-07 03:21:21

Run sshd in debug and then connect:

/usr/sbin/sshd -d

Try using -dd or -ddd if you can't see anything useful at level 1.

curtisleebolin · 2014-09-07 03:40:09

I have been stumped for more than a day, but it just dawned on me when I released it was a PAM problem.

In my airootfs/root/customize_airootfs.sh file I simply changed zsh to bash because I like bash.

usermod -s /usr/bin/bash root

I remember over a year ago having a similar PAM problem. For some reason PAM doesn't like /usr/bin/bash set as the shell. I had to change it to /bin/bash.

usermod -s /bin/bash root

Now everything works fine. I'll mark this as solved and make a new post asking about PAM.

fukawi2 · 2014-09-07 07:08:28

curtisleebolin wrote:

For some reason PAM doesn't like /usr/bin/bash set as the shell. I had to change it to /bin/bash.

Probably because /usr/bin/bash isn't a valid shell in /etc/shells

curtisleebolin · 2014-09-07 07:17:22

fukawi2 wrote:

Probably because /usr/bin/bash isn't a valid shell in /etc/shells

I opened a new thread for the PAM question https://bbs.archlinux.org/viewtopic.php?id=186720

Even if /usr/bin/bash is added to /etc/shells, it still fails.

Arch Linux

#1 2014-09-07 02:45:36

[SOLVED] SSH problem with archiso with sshd enabled for system mainten

#2 2014-09-07 03:17:02

Re: [SOLVED] SSH problem with archiso with sshd enabled for system mainten

#3 2014-09-07 03:21:21

Re: [SOLVED] SSH problem with archiso with sshd enabled for system mainten

#4 2014-09-07 03:40:09

Re: [SOLVED] SSH problem with archiso with sshd enabled for system mainten

#5 2014-09-07 07:08:28

Re: [SOLVED] SSH problem with archiso with sshd enabled for system mainten

#6 2014-09-07 07:17:22

Re: [SOLVED] SSH problem with archiso with sshd enabled for system mainten

Board footer