[Solved]sshd fails to start after update

Zapman · 2017-07-05 20:05:31

Hello, sshd no longer starts and i get segmentation fault when i try to run it manually.
Problem appeared after issuing pacman -Syu :

One thing that caught my eye is : Start request repeated too quickly

How can i fix this? Thanks a ton guyz..

sshd status wrote:

sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: disabled)
Active: failed (Result: core-dump) since Wed 2017-07-05 22:57:52 EEST; 1min 36s ago
Process: 412 ExecStart=/usr/bin/sshd -D (code=dumped, signal=SEGV)
Main PID: 412 (code=dumped, signal=SEGV)
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Main process exited, code=dumped, status=11/SEGV
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Unit entered failed state.
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Failed with result 'core-dump'.
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Service hold-off time over, scheduling restart.
Ιούλ 05 22:57:52 archlinux systemd[1]: Stopped OpenSSH Daemon.
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Start request repeated too quickly.
Ιούλ 05 22:57:52 archlinux systemd[1]: Failed to start OpenSSH Daemon.
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Unit entered failed state.
Ιούλ 05 22:57:52 archlinux systemd[1]: sshd.service: Failed with result 'core-dump'.

ssh -ddd wrote:

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 399
debug2: parse_server_config: config /etc/ssh/sshd_config len 399
debug3: /etc/ssh/sshd_config:13 setting Port 22
debug3: /etc/ssh/sshd_config:15 setting ListenAddress 0.0.0.0
debug3: /etc/ssh/sshd_config:16 setting ListenAddress ::
debug3: /etc/ssh/sshd_config:21 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: /etc/ssh/sshd_config:23 setting AllowUsers lab
debug3: /etc/ssh/sshd_config:39 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:43 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: /etc/ssh/sshd_config:59 setting PasswordAuthentication no
debug3: /etc/ssh/sshd_config:63 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:84 setting UsePAM yes
debug3: /etc/ssh/sshd_config:93 setting PrintMotd no
debug3: /etc/ssh/sshd_config:112 setting Subsystem sftp /usr/lib/ssh/sftp-server
debug1: sshd version OpenSSH_7.5, OpenSSL 1.1.0f 25 May 2017
(segmentation fault)

Last edited by Zapman (2017-07-07 20:45:21)

jasonwryan · 2017-07-05 20:35:30

Please paste (using a paste service) your full pacman log.

Zapman · 2017-07-06 08:38:13

Here it is

seth · 2017-07-06 12:08:18

debug3: /etc/ssh/sshd_config:112 setting Subsystem sftp    /usr/lib/ssh/sftp-server
debug1: sshd version OpenSSH_7.5, OpenSSL 1.1.0f  25 May 2017
(segmentation fault)

[2017-07-04 17:17] [ALPM] warning: /etc/ssh/sshd_config installed as /etc/ssh/sshd_config.pacnew
[2017-07-04 17:17] [ALPM] upgraded openssh (7.3p1-1 -> 7.5p1-2)

Out of curiosity: Why is there

[2014-03-19 14:53] [PACMAN] Running 'pacman -S yaourt'
[2014-03-19 15:10] [PACMAN] Running 'pacman -Sy yaourt'
[2014-03-19 15:10] [PACMAN] synchronizing package lists
[2014-03-19 15:11] [PACMAN] installed yajl (2.0.4-2)
[2014-03-19 15:11] [PACMAN] installed package-query (1.2-2)
[2014-03-19 15:11] [PACMAN] installed yaourt (1.3-1)

Did arch ever have yaourt in the repos??

mrunion · 2017-07-06 15:43:13

seth wrote:

Did arch ever have yaourt in the repos??

User probably has added the archlinuxfr repository in the past I would think.

Zapman · 2017-07-06 16:05:09

Yes, thats right. Was it the cause of the problem? What should i do?

seth · 2017-07-06 16:13:35

I'd rather look at the deviating sshd_config file (and hope that you removed the archlinuxfr repo ... ;-)

Zapman · 2017-07-06 17:04:01

Yes archlinuxfr is long gone. sshd_config

Last edited by Zapman (2017-07-06 17:24:44)

seth · 2017-07-06 19:09:42

Nope, looks sane.
Try to gdb sshd.

Zapman · 2017-07-06 19:40:54

gdb wrote:

program received signal SIGSEGV
in __memset_ia32() from /usr/lib/lib.c.so.6

Thats all i got. i don't know much about gdb though

seth · 2017-07-06 19:46:02

try "bt" (as in "backtrace)

Zapman · 2017-07-06 20:31:36

gdb wrote:

program received signal SIGSEGV
#0 in __memset_ia32() from /usr/lib/lib.c.so.6
#1 in __explicit_bzero_chk_internal () from /usr/lib/lib.c.so.6
#2 in main()

seth · 2017-07-06 20:36:29

google on "sshd __explicit_bzero_chk_internal segfault" :
https://bugs.launchpad.net/ubuntu/+sour … ug/1268719
https://bugs.launchpad.net/ubuntu/+sour … bug/371659
https://bugs.debian.org/cgi-bin/bugrepo … bug=764799

The first two are about messup with your network/hosts and the 3rd about a missing host key.

Zapman · 2017-07-06 21:07:39

ssh-keygen -A solved nothing

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet 192.168.1.20/24 brd 192.168.1.255 scope global enp0s3
valid_lft forever preferred_lft forever

I dont know what else to do.. i give up. Thank you for your help mate, much appreciated

seth · 2017-07-06 21:34:49

Maybe the present /etc/ssh/ssh_host_ed25519_key is garbled?

wc /etc/hosts

Zapman · 2017-07-07 07:46:37

wc /etc/hosts wrote:

9 23 195 /etc/hosts

I also tried pacman -Rsn openssh and removed /etc/ssh. No luck..

jasonwryan · 2017-07-07 07:49:37

Please paste the output of `pacman -Qi openssh && type ssh`.

Zapman · 2017-07-07 08:23:37

Name : openssh
Version : 7.5p1-2
Description : Free version of the SSH connectivity tools
Architecture : i686
URL : https://www.openssh.com/portable.html
Licenses : custom:BSD
Groups : None
Provides : None
Depends On : krb5 openssl libedit ldns
Optional Deps : xorg-xauth: X11 forwarding [installed]
x11-ssh-askpass: input passphrase in X
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 5.49 MiB
Packager : Gaetan Bisson <bisson@archlinux.org>
Build Date : Mon 20 Mar 2017 11:40:50 PM EET
Install Date : Fri 07 Jul 2017 11:08:37 AM EEST
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
ssh is /usr/bin/ssh

seth · 2017-07-07 13:55:12

There're only 3 explicit_bzero calls in sshd.c and only one directly in the main() routine. It deals with the privsep password.
Did you setup a sshd user to run sshd under this UID?

Zapman · 2017-07-07 14:44:02

no i did no such thing. At least i dont recall doing it. should i make a user?
edit: confirmed no such user exists

Last edited by Zapman (2017-07-07 14:47:19)

seth · 2017-07-07 14:48:44

grep ssh /etc/passwd

No, but I wonder whether it really falls flat nuking a static array or there's sth. so badly broken that it enters the false code branch...
gcc7+ia32 related issue?

Zapman · 2017-07-07 14:55:16

any alternatives to openssh???

seth · 2017-07-07 15:05:44

I'd not ignore that - the problem will in this case rather not be in sshd itself.
That said: just tried. Runs fine on a i686 CPU - here.

=> "valgrind sshd"

Zapman · 2017-07-07 16:59:42

valgrind --leak-check=full wrote:

==507== Memcheck, a memory error detector
==507== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==507== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==507== Command: /usr/bin/sshd
==507==
==507== Warning: invalid file descriptor 1024 in syscall close()
==507== Warning: invalid file descriptor 1025 in syscall close()
==507== Warning: invalid file descriptor 1026 in syscall close()
==507== Warning: invalid file descriptor 1027 in syscall close()
==507== Use --log-fd=<number> to select an alternative log fd.
==507== Warning: invalid file descriptor 1028 in syscall close()
==507== Warning: invalid file descriptor 1029 in syscall close()
==507== Warning: invalid file descriptor 1030 in syscall close()
==507==
==507== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==507== Bad permissions for mapped region at address 0x5379D53
==507== at 0x4CC158E: __GI_memset (in /usr/lib/libc-2.25.so)
==507== by 0x4D45F07: __explicit_bzero_chk (in /usr/lib/libc-2.25.so)
==507== by 0x10F268: main (in /usr/bin/sshd)
==507==
==507== HEAP SUMMARY:
==507== in use at exit: 42,289 bytes in 2,685 blocks
==507== total heap usage: 2,822 allocs, 137 frees, 106,399 bytes allocated
==507==
==507== 634 (80 direct, 554 indirect) bytes in 1 blocks are definitely lost in loss record 667 of 683
==507== at 0x482F556: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==507== by 0x188D97: ??? (in /usr/bin/sshd)
==507== by 0x10EABE: main (in /usr/bin/sshd)
==507==
==507== LEAK SUMMARY:
==507== definitely lost: 80 bytes in 1 blocks
==507== indirectly lost: 554 bytes in 19 blocks
==507== possibly lost: 0 bytes in 0 blocks
==507== still reachable: 41,655 bytes in 2,665 blocks
==507== suppressed: 0 bytes in 0 blocks
==507== Reachable blocks (those to which a pointer was found) are not shown.

interesting indeed

seth · 2017-07-07 19:30:03

Bad permissions for mapped region at address 0x5379D53

That's a bad heap access and that cries it's "explicit_bzero(privsep_pw->pw_passwd" and that means "getpwnam(SSH_PRIVSEP_USER)" is not null ...

And ("tada") the reason is that arch compiles "nobody" as the privsep_user ...

sudo grep nobody  /etc/passwd /etc/shadow

The passwd match is safe to post, the shadow output should look like

nobody:x:14871::::::

If it does not, figure why. Do not post password hashes!

Arch Linux

#1 2017-07-05 20:05:31

[Solved]sshd fails to start after update

#2 2017-07-05 20:35:30

Re: [Solved]sshd fails to start after update

#3 2017-07-06 08:38:13

Re: [Solved]sshd fails to start after update

#4 2017-07-06 12:08:18

Re: [Solved]sshd fails to start after update

#5 2017-07-06 15:43:13

Re: [Solved]sshd fails to start after update

#6 2017-07-06 16:05:09

Re: [Solved]sshd fails to start after update

#7 2017-07-06 16:13:35

Re: [Solved]sshd fails to start after update

#8 2017-07-06 17:04:01

Re: [Solved]sshd fails to start after update

#9 2017-07-06 19:09:42

Re: [Solved]sshd fails to start after update

#10 2017-07-06 19:40:54

Re: [Solved]sshd fails to start after update

#11 2017-07-06 19:46:02

Re: [Solved]sshd fails to start after update

#12 2017-07-06 20:31:36

Re: [Solved]sshd fails to start after update

#13 2017-07-06 20:36:29

Re: [Solved]sshd fails to start after update

#14 2017-07-06 21:07:39

Re: [Solved]sshd fails to start after update

#15 2017-07-06 21:34:49

Re: [Solved]sshd fails to start after update

#16 2017-07-07 07:46:37

Re: [Solved]sshd fails to start after update

#17 2017-07-07 07:49:37

Re: [Solved]sshd fails to start after update

#18 2017-07-07 08:23:37

Re: [Solved]sshd fails to start after update

#19 2017-07-07 13:55:12

Re: [Solved]sshd fails to start after update

#20 2017-07-07 14:44:02

Re: [Solved]sshd fails to start after update

#21 2017-07-07 14:48:44

Re: [Solved]sshd fails to start after update

#22 2017-07-07 14:55:16

Re: [Solved]sshd fails to start after update

#23 2017-07-07 15:05:44

Re: [Solved]sshd fails to start after update

#24 2017-07-07 16:59:42

Re: [Solved]sshd fails to start after update

#25 2017-07-07 19:30:03

Re: [Solved]sshd fails to start after update

Board footer