Arch Linux

wantilles

Member

From: Athens - Greece

Registered: 2007-03-29

Posts: 327

[SOLVED] [BUG] ssh does not work after today's package update

I have a computer running arch amd64, on a server role.

After today's package update, it does not accept ssh connections.

Here is the relevant pacman.log:

[2009-10-28 12:31] starting full system upgrade
[2009-10-28 12:31] upgraded kernel-headers (2.6.30.5-1 -> 2.6.31.4-1)
[2009-10-28 12:31] Generating locales...
[2009-10-28 12:31]   el_GR.UTF-8... done
[2009-10-28 12:31]   el_GR.ISO-8859-7... done
[2009-10-28 12:31]   el_GR.ISO-8859-7@euro... done
[2009-10-28 12:31]   en_US.UTF-8... done
[2009-10-28 12:31]   en_US.ISO-8859-1... done
[2009-10-28 12:31] Generation complete.
[2009-10-28 12:31] upgraded glibc (2.10.1-4 -> 2.10.1-5)
[2009-10-28 12:31] upgraded binutils (2.19.1-5 -> 2.20-1)
[2009-10-28 12:31] upgraded openntpd (3.9p1-7 -> 3.9p1-8)
[2009-10-28 12:31] upgraded python (2.6.3-2 -> 2.6.4-1)
[2009-10-28 12:31] upgraded smartmontools (5.38-1 -> 5.38-2)

What is going on?

None of these packages seem related to ssh.

PS: The network is working normally because:

- I can ping the server
- The permanet nfs shares that I import from there, at my fstab, have been mounted normally and are working.

Last edited by wantilles (2009-10-28 17:05:13)

wantilles

Member

From: Athens - Greece

Registered: 2007-03-29

Posts: 327

Re: [SOLVED] [BUG] ssh does not work after today's package update

I have isolated the problem.

The sshd daemon fails to start:

:: Starting Secure Shell Daemon                                          [BUSY]
Missing privilege separation directory: /var/empty
                                                                         [FAIL]

The directory did not exist.

I created it manually and now it starts and works.

Is this a bug created by one of the above-mentioned today's packages?

Should I file a bug report?

Last edited by wantilles (2009-10-28 15:31:03)

Allan

Pacman

From: Brisbane, AU

Registered: 2007-06-09

Posts: 11,472

Website

Re: [SOLVED] [BUG] ssh does not work after today's package update

> pacman -Ql | grep var/empty
openssh /var/empty/

weird...

---

PGP Key: F99FFE0FEAE999BD | I develop pacman - buy me a drink!

Ogion

Member

From: Germany

Registered: 2007-12-11

Posts: 367

Re: [SOLVED] [BUG] ssh does not work after today's package update

$ p -Ql | grep var/empty
openntpd /var/empty/
openssh /var/empty/

Well, this is what it says on my machine, so maybe the openntpd update did something to that file?

Ogion

---

(my-dotfiles)
"People willing to trade their freedom for temporary security deserve neither and will lose both." - Benjamin Franklin
"Enlightenment is man's leaving his self-caused immaturity." - Immanuel Kant

Harey

Member

From: Bavaria, Germany

Registered: 2007-03-24

Posts: 372

Re: [SOLVED] [BUG] ssh does not work after today's package update

$ p -Ql | grep var/empty
openntpd /var/empty/
openssh /var/empty/

Well, this is what it says on my machine, so maybe the openntpd update did something to that file?

Ogion

Yep. I guess the post_install of the openntpd package is deleting the directory /var/empty by invoking useradd. If you are upgrading via ssh then you should do

mkdir /var/empty

directly after the upgrade or you will be locked out

Harvey

---

Linux is like a wigwam: No Gates, no Windows and an Apache inside

wantilles

Member

From: Athens - Greece

Registered: 2007-03-29

Posts: 327

Re: [SOLVED] [BUG] ssh does not work after today's package update

If you are upgrading via ssh then you should do

mkdir /var/empty

directly after the upgrade or you will be locked out

I was updating the server via ssh.

I always do, since this is a remote, headless server, connected by wired gigabit ethernet.

wantilles

Member

From: Athens - Greece

Registered: 2007-03-29

Posts: 327

Re: [SOLVED] [BUG] ssh does not work after today's package update

I filed a bug report:

http://bugs.archlinux.org/task/16886

toxygen

Member

Registered: 2008-08-22

Posts: 713

Re: [SOLVED] [BUG] ssh does not work after today's package update

wish i had seen this post before i left home, cant connect now
voted +1 for your bug, though the solution is simple (if you're on the computer being affected).

---

"I know what you're thinking, 'cause right now I'm thinking the same thing. Actually, I've been thinking it ever since I got here:
Why oh why didn't I take the BLUE pill?"

ngoonee

Forum Fellow

From: Between Thailand and Singapore

Registered: 2009-03-17

Posts: 7,358

Re: [SOLVED] [BUG] ssh does not work after today's package update

Ah, just got hit by this. Thankfully the computer is right here on my table, wouldn't have liked this to happen to a computer in the next time-zone.

---

Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

dw

Member

From: Vienna, Austria

Registered: 2006-11-25

Posts: 160

Re: [SOLVED] [BUG] ssh does not work after today's package update

Ah, just got hit by this. Thankfully the computer is right here on my table, wouldn't have liked this to happen to a computer in the next time-zone.

you're lucky. I also got hit by this, but on a headless remote server far away.

any hints on what I can do without physical access? nothing I guess, am I right?

dr.zentech

Member

Registered: 2009-05-27

Posts: 25

Re: [SOLVED] [BUG] ssh does not work after today's package update

I just had this hit three of my machines... two are located here in my basement, so no biggy. The third is unfortunately six hours away from here and a headless machine, which makes walking my mother-in-law through the steps quite tricky.

Bleh

bodyhead

Member

Registered: 2009-10-25

Posts: 6

Re: [SOLVED] [BUG] ssh does not work after today's package update

I'm also affected.  +1 for bug report.

Youdaman

Member

From: Sydney, Australia

Registered: 2006-01-14

Posts: 41

Re: [SOLVED] [BUG] ssh does not work after today's package update

woah, close one -- I noticed my winscp sftp session to my archlinux box had died and reconnection kept failing... lucky i had a terminal open still! mkdir /var/empty indeed fixed it:

[root@hax youdaman]# /etc/rc.d/sshd restart
:: Stopping Secure Shell Daemon                                          [FAIL]
:: Starting Secure Shell Daemon                                          [BUSY] Missing privilege separation directory: /var/empty
                                                                         [FAIL]
[root@hax youdaman]# mkdir /var/empty
[root@hax youdaman]# /etc/rc.d/sshd restart
:: Stopping Secure Shell Daemon                                          [FAIL]
:: Starting Secure Shell Daemon                                          [DONE]

btw, I'd just upgraded openntpd also, so I can also confirm it was the problem as above. From /var/log/pacman.log:

[2009-10-30 15:49] upgraded openntpd (3.9p1-7 -> 3.9p1-9)

Last edited by Youdaman (2009-10-30 05:32:14)

sand_man

Member

From: Australia

Registered: 2008-06-10

Posts: 2,164

Re: [SOLVED] [BUG] ssh does not work after today's package update

/me looks at Allan...

Last edited by sand_man (2009-10-30 06:04:45)

---

rpgcyco

Member

From: Australia

Registered: 2005-09-27

Posts: 74

Re: [SOLVED] [BUG] ssh does not work after today's package update

This got me as well, but I luckily I had VNC access, so all was well.

colbert

Member

Registered: 2007-12-16

Posts: 809

Re: [SOLVED] [BUG] ssh does not work after today's package update

Argh, headless server as well. I've been getting an "ssh_exchange_identification" error. With -v I see it connects, key is passed and all, but then gives that. In another thread I saw that I need to add my host to the /etc/hosts.allow of the server, but I've been connecting for ages just fine, would that have been affected by the ssh upgrade? Or is it this /var/empty issue?

ataraxia

Member

From: Pittsburgh

Registered: 2007-05-06

Posts: 1,553

Re: [SOLVED] [BUG] ssh does not work after today's package update

Argh, headless server as well. I've been getting an "ssh_exchange_identification" error. With -v I see it connects, key is passed and all, but then gives that. In another thread I saw that I need to add my host to the /etc/hosts.allow of the server, but I've been connecting for ages just fine, would that have been affected by the ssh upgrade? Or is it this /var/empty issue?

That's the /var/empty issue. That's how it fails if you lose /var/empty while sshd is already running.

colbert

Member

Registered: 2007-12-16

Posts: 809

Re: [SOLVED] [BUG] ssh does not work after today's package update

Okay, well I plan on bringing the box up from basement to my room and plugging into the monitor so I can fix what needs fixin'. What exactly do I do? Just mkdir /var/empty, or?

ataraxia

Member

From: Pittsburgh

Registered: 2007-05-06

Posts: 1,553

Re: [SOLVED] [BUG] ssh does not work after today's package update

That should be enough. If you upgrade openssh, the new rc script for it makes sure the directory is there on each start.