You are not logged in.
Our remote server (running 2.6.11.10) rebooted unexpectedly this morning.
I checked the logs and found that we did get hammered with breakin attempts this morning for about 15 minutes, but this stopped about an hour-and-a-half before the reboot.
Almost immediately before the reboot, there is a strange PAM error in the log.
Aug 15 07:44:23 server sshd[1573]: error: Could not get shadow information for NOUSER
Aug 15 07:44:24 server sshd[1576]: error: Could not get shadow information for NOUSER
Aug 15 07:44:24 server sshd[1579]: error: Could not get shadow information for NOUSER
Aug 15 09:26:06 server login(pam_unix)[2885]: check pass; user unknown
Aug 15 09:26:21 server login(pam_unix)[2885]: bad username [^[0000]
To the best of my knowledge, no one was able to login. Root login over SSH is disabled, and there are only two users that even have a shell. I did have an open ssh window from a different machine, and here is what it reported when it rebooted:
Broadcast message from root (console) (Mon Aug 15 09:26:46 2005):
The system is going down for reboot NOW!
Does this mean the shutdown was issued from the console? Have I been hacked?
Offline
Hmmm try chrootkit and all that jazz - keep it off the network for now...
verify that no passwords were changed (when that happened to me, they gained root access and change root's passwd)... the "bad username" part looks funky because there's escape characters in it - so it may be someone exploiting an odd ssh hole....
Offline
These things (ssh login attempts and reboot) might not be connected.
As the ssh message suggests someone at the server could just press ctrl alt del accidentally (wrong keyboard switch?) :-)
Want some ssh login attempts?
root ssh:notty 211.233.9.185 Tue Aug 16 14:01 - 14:01 (00:00)
demouser ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
demo ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
demo ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
news ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
ftp ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
postgres ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
postgres ssh:notty c-24-20-206-160. Tue Aug 16 03:54 - 03:54 (00:00)
postmast ssh:notty c-24-20-206-160. Tue Aug 16 03:53 - 03:53 (00:00)
postgres ssh:notty c-24-20-206-160. Tue Aug 16 03:53 - 03:53 (00:00)
demo ssh:notty c-24-20-206-160. Tue Aug 16 03:53 - 03:53 (00:00)
pooola ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
google ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
mysq ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
oracle ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
ok ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
webdesin ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
manager ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
master ssh:notty 211.233.13.234 Mon Aug 15 16:14 - 16:14 (00:00)
web ssh:notty 211.233.13.234 Mon Aug 15 16:13 - 16:13 (00:00)
Enjoy :-)
BTW They are now more creative and they started to use different dictionaries now.
Offline
you dont even want to see my ssh logs.
o.0
"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍
Offline
These things (ssh login attempts and reboot) might not be connected.
As the ssh message suggests someone at the server could just press ctrl alt del accidentally (wrong keyboard switch?) :-)
That's very possible - this machine is a colo and I suspect that someone there may have issued a ctrl-alt-del, thinking they were hooked up to a different machine (only one keyboard and monitor per rack, controlled via a switcher.)
If so, is there anyway to get Arch to ignore a ctrl-alt-del? Or is that possible through the BIOS?
Offline
comment out
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
in /etc/inittab should work.
"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍
Offline
comment out
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
in /etc/inittab should work.
Fantastic! Just tried it out on a test box.
Thanks to all! I'm about 99.9% sure now that somebody at the datacenter where we colo rebooted us by mistake, but that won't be happening again.
Offline