Unscheduled reboot after breaking attempt

marcob · 2005-08-15 17:13:44

Our remote server (running 2.6.11.10) rebooted unexpectedly this morning.

I checked the logs and found that we did get hammered with breakin attempts this morning for about 15 minutes, but this stopped about an hour-and-a-half before the reboot.

Almost immediately before the reboot, there is a strange PAM error in the log.

Aug 15 07:44:23 server sshd[1573]: error: Could not get shadow information for NOUSER
Aug 15 07:44:24 server sshd[1576]: error: Could not get shadow information for NOUSER
Aug 15 07:44:24 server sshd[1579]: error: Could not get shadow information for NOUSER
Aug 15 09:26:06 server login(pam_unix)[2885]: check pass; user unknown
Aug 15 09:26:21 server login(pam_unix)[2885]: bad username [^[0000]

To the best of my knowledge, no one was able to login. Root login over SSH is disabled, and there are only two users that even have a shell. I did have an open ssh window from a different machine, and here is what it reported when it rebooted:

Broadcast message from root (console) (Mon Aug 15 09:26:46 2005):

The system is going down for reboot NOW!

Does this mean the shutdown was issued from the console? Have I been hacked?

phrakture · 2005-08-15 18:01:12

Hmmm try chrootkit and all that jazz - keep it off the network for now...
verify that no passwords were changed (when that happened to me, they gained root access and change root's passwd)... the "bad username" part looks funky because there's escape characters in it - so it may be someone exploiting an odd ssh hole....

lanrat · 2005-08-16 13:20:24

These things (ssh login attempts and reboot) might not be connected.
As the ssh message suggests someone at the server could just press ctrl alt del accidentally (wrong keyboard switch?) :-)

Want some ssh login attempts?

root     ssh:notty    211.233.9.185    Tue Aug 16 14:01 - 14:01  (00:00)
demouser ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
demo     ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
demo     ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
news     ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
ftp      ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
postgres ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
postgres ssh:notty    c-24-20-206-160. Tue Aug 16 03:54 - 03:54  (00:00)
postmast ssh:notty    c-24-20-206-160. Tue Aug 16 03:53 - 03:53  (00:00)
postgres ssh:notty    c-24-20-206-160. Tue Aug 16 03:53 - 03:53  (00:00)
demo     ssh:notty    c-24-20-206-160. Tue Aug 16 03:53 - 03:53  (00:00)
pooola   ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
google   ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
mysq     ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
oracle   ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
ok       ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
webdesin ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
manager  ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
master   ssh:notty    211.233.13.234   Mon Aug 15 16:14 - 16:14  (00:00)
web      ssh:notty    211.233.13.234   Mon Aug 15 16:13 - 16:13  (00:00)

Enjoy :-)
BTW They are now more creative and they started to use different dictionaries now.

cactus · 2005-08-16 14:50:52

you dont even want to see my ssh logs.
o.0

marcob · 2005-08-16 20:47:02

lanrat wrote:

These things (ssh login attempts and reboot) might not be connected.
As the ssh message suggests someone at the server could just press ctrl alt del accidentally (wrong keyboard switch?) :-)

That's very possible - this machine is a colo and I suspect that someone there may have issued a ctrl-alt-del, thinking they were hooked up to a different machine (only one keyboard and monitor per rack, controlled via a switcher.)

If so, is there anyway to get Arch to ignore a ctrl-alt-del? Or is that possible through the BIOS?

cactus · 2005-08-16 20:53:26

comment out

ca::ctrlaltdel:/sbin/shutdown -t3 -r now

in /etc/inittab should work.

marcob · 2005-08-16 22:14:17

cactus wrote:

comment out
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
in /etc/inittab should work.

Fantastic! Just tried it out on a test box.

Thanks to all! I'm about 99.9% sure now that somebody at the datacenter where we colo rebooted us by mistake, but that won't be happening again.

Arch Linux

#1 2005-08-15 17:13:44

Unscheduled reboot after breaking attempt

#2 2005-08-15 18:01:12

Re: Unscheduled reboot after breaking attempt

#3 2005-08-16 13:20:24

Re: Unscheduled reboot after breaking attempt

#4 2005-08-16 14:50:52

Re: Unscheduled reboot after breaking attempt

#5 2005-08-16 20:47:02

Re: Unscheduled reboot after breaking attempt

#6 2005-08-16 20:53:26

Re: Unscheduled reboot after breaking attempt

#7 2005-08-16 22:14:17

Re: Unscheduled reboot after breaking attempt

Board footer