You are not logged in.

#1 2017-05-05 17:17:49

archtastic
Member
Registered: 2017-05-05
Posts: 1

debugging "uninterruptible sleep errors" on many systemd processes

I have been running my arch box as a home media server for about a couple of years. I usually leave it running 24/7.

Every week or so, some of the key services stop responding. These include (sshd, plexmediaserver,samba) etc.

Oddly enough, apache and webmin still work when this happens, so I can use webmin to run commands on the server even though ssh doesn't work. When I try to stop/restart these processes using systemd, nothing is fixed. Even manually killing with "kill -9" does not seem to work. When I take a look at the processes using ps, I get a ton of these "D" statuses, which apparently mean that the process cannot be killed. See below.

> ps aux | grep sshd
root       971  0.0  0.1  38300  4420 ?        Ds   12:59   0:00 /usr/bin/sshd -D
root      1318  0.0  0.0  20392  2844 ?        S    13:08   0:00 sh -c (ps aux | grep sshd) 2>&1
root      1319  0.0  0.0  20392   300 ?        S    13:08   0:00 sh -c (ps aux | grep sshd) 2>&1
root      1321  0.0  0.0   9024   880 ?        S    13:08   0:00 grep sshd
root     29017  0.0  0.1  38300  4500 ?        Ds   11:54   0:00 /usr/bin/sshd -D -R
root     29146  0.0  0.1  38300  4304 ?        Ds   11:55   0:00 /usr/bin/sshd -D -R
root     29193  0.0  0.0  38300  2268 ?        Ds   11:56   0:00 /usr/bin/sshd -D -R
root     29570  0.0  0.1  38300  4528 ?        Ds   12:02   0:00 /usr/bin/sshd -D -R
root     29962  0.0  0.1  38300  4284 ?        Ds   12:09   0:00 /usr/bin/sshd -D -R
root     30319  0.0  0.0  38300  2256 ?        Ds   12:15   0:00 /usr/bin/sshd -D -R
root     30706  0.0  0.0  38300  2228 ?        Ds   12:21   0:00 /usr/bin/sshd -D -R
root     30717  0.0  0.1  38300  4388 ?        Ds   12:22   0:00 /usr/bin/sshd -D -R
root     31138  0.0  0.1  38300  4452 ?        Ds   12:28   0:00 /usr/bin/sshd -D -R
root     31226  0.0  0.1  38300  4372 ?        Ds   12:30   0:00 /usr/bin/sshd -D -R
root     31348  0.0  0.1  38300  4308 ?        Ds   12:31   0:00 /usr/bin/sshd -D -R
root     31584  0.0  0.1  38300  4420 ?        Ds   12:34   0:00 /usr/bin/sshd -D -R
root     31747  0.0  0.0  38300  2268 ?        Ds   12:36   0:00 /usr/bin/sshd -D -R
root     32126  0.0  0.1  38300  4380 ?        Ds   12:41   0:00 /usr/bin/sshd -D -R
root     32491  0.0  0.1  38300  4368 ?        Ds   12:47   0:00 /usr/bin/sshd -D -R
root     32638  0.0  0.1  38300  4500 ?        Ds   12:51   0:00 /usr/bin/sshd -D -R
root     32702  0.0  0.0  38300  2252 ?        Ds   12:52   0:00 /usr/bin/sshd -D -R
root     32766  0.0  0.0  38300  2280 ?        Ds   12:53   0:00 /usr/bin/sshd -D

When this happens, I am not even able to restart the machine by typing "shutdown -r now". The only thing that fixes this is a hard reset (holding down power button). This has happened about every 7-14 days since I first started running the machine, and I am getting pretty frustrated, so I appreciate any help you can provide.

Offline

#2 2017-05-06 07:50:22

Starfish
Member
From: Germany
Registered: 2015-10-21
Posts: 134

Re: debugging "uninterruptible sleep errors" on many systemd processes

Do you regularly upgrade your system? I could imagine that if you do, and you leave it running without restarting, sooner or later there will be discrepancies between the old packages in your RAM and the new ones on your hard disk. In particular, one of your daemons could dynamically load a library which has changed its interface since the last update and thus crash.
I have never managed a server, so this is a wild guess.


"Yesterday is history, tomorrow is a mystery, but today is a gift. That is why it is called the present." - Master Oogway

Offline

#3 2017-05-06 08:19:38

seth
Member
From: Won't reply 2 private help req
Registered: 2012-09-03
Posts: 78,891

Re: debugging "uninterruptible sleep errors" on many systemd processes

The processes wait for some I/O (ioctl) response and the pattern says: network issue.
The state is not interruptable, ie. not killable either, that's "normal" and to protect data integrity.

Lookup your dmesg, the network device(s?) might take a hiccup - "once a week" sounds like induced by a router reset?

Offline

Board footer

Powered by FluxBB