You are not logged in.

#1 2022-11-16 08:08:21

marlemion
Member
Registered: 2022-11-16
Posts: 6

[SOLVED] Random shutdowns preferably at night

Dear all,

I have several arch installs running, in particular one of them being the main server of our house hosting quite a few services such as nextcloud docker, http server, NFS and SMB file servers, Lidarr, Radarr, Sonarr, MySQL DB, vdr etc.

This setup has been installed many years ago and has been migrated from a few hardwares already. A few days back I encountered a spontaneous shutdown at night (3am CET). As I wanted to change the hardware anyway I took the opportunity and migrated the system from the original hardware (ASRock BeeBox N3160 w. 8GB RAM) to a new system (Intel NUC8i3BEH w. 40GB RAM). The remaining hardware was the same, i.e. 2TB internal, 2*2TB USB, two DVB-S USB Receiver. Still, the shutdowns persist.

last -x | egrep -E 'shutdown|reboot' | head -n 40

reboot   system boot  6.0.8-arch1-1    Wed Nov 16 06:36   still running
shutdown system down  6.0.8-arch1-1    Wed Nov 16 04:09 - 06:36  (02:26)
reboot   system boot  6.0.8-arch1-1    Tue Nov 15 18:05 - 04:09  (10:03)
shutdown system down  6.0.8-arch1-1    Tue Nov 15 11:15 - 18:05  (06:50)
reboot   system boot  6.0.8-arch1-1    Tue Nov 15 06:14 - 11:15  (05:00)
shutdown system down  6.0.8-arch1-1    Tue Nov 15 05:51 - 06:14  (00:23)
reboot   system boot  6.0.8-arch1-1    Mon Nov 14 09:27 - 05:51  (20:23)
shutdown system down  6.0.6-arch1-1    Mon Nov 14 09:26 - 09:27  (00:00)
reboot   system boot  6.0.6-arch1-1    Mon Nov 14 06:18 - 09:26  (03:08)
shutdown system down  6.0.6-arch1-1    Mon Nov 14 03:36 - 06:18  (02:41)

LOTS OF INTENDED REBOOTS BEFORE DUE TO MIGRATION

reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:25 - 03:36  (14:11)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 13:22 - 13:25  (00:02)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:20 - 13:22  (00:01)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 13:18 - 13:20  (00:02)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:16 - 13:18  (00:01)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 13:12 - 13:16  (00:03)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:11 - 13:12  (00:01)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 13:10 - 13:11  (00:00)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:08 - 13:10  (00:02)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 13:07 - 13:08  (00:00)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 13:05 - 13:07  (00:01)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 12:57 - 13:07  (00:09)
shutdown system down  6.0.6-arch1-1    Sun Nov 13 12:56 - 12:57  (00:01)
reboot   system boot  6.0.6-arch1-1    Sun Nov 13 12:28 - 12:56  (00:27)

NEW HARDWARE

shutdown system down  6.0.6-arch1-1    Sat Nov 12 18:43 - 12:28  (17:45)
reboot   system boot  6.0.6-arch1-1    Sat Nov 12 08:03 - 18:43  (10:39)
shutdown system down  6.0.6-arch1-1    Sat Nov 12 04:49 - 08:03  (03:13)
reboot   system boot  6.0.6-arch1-1    Fri Nov 11 18:55 - 04:49  (09:53)
shutdown system down  6.0.6-arch1-1    Fri Nov 11 03:31 - 18:55  (15:23)
reboot   system boot  6.0.6-arch1-1    Mon Oct 31 18:28 - 03:31 (10+09:03)

NEW KERNEL

shutdown system down  5.18.16-arch1-1  Mon Oct 31 18:27 - 18:28  (00:00)

As you can see, I have performed an update before the first reboot at 3:31am Nov 11 occurred (new kernel). From there on I had constant shutdowns every night in between 3 and 6 am with one exception: Nov 15 at 11:15am. It is completely weird. I have disabled all cronjobs - no effect. The logs are not helpful. Excerpts:

Nov 16 04:01:01 vdr CROND[32828]: (root) CMD (run-parts /etc/cron.hourly)
Nov 16 04:01:01 vdr CROND[32827]: (root) CMDEND (run-parts /etc/cron.hourly)
Nov 16 04:03:07 vdr Radarr[2511]: SQLite error (5): database is locked in "PRAGMA cache_size=-20000"
Nov 16 04:03:32 vdr vdr[478]: [478] ERROR: no OSD provider available - using dummy OSD!
Nov 16 04:08:33 vdr systemd-logind[471]: The system will power off now!
Nov 16 04:08:33 vdr systemd-logind[471]: System is powering down.
Nov 15 11:12:02 vdr mono[2403]: [Info] RssSyncService: Starting RSS Sync
Nov 15 11:12:02 vdr jackett[448]: 11-15 11:12:02 Info Torznab search in xxx.xxx => Found 13 releases (from cache)
Nov 15 11:12:02 vdr jackett[448]: 11-15 11:12:02 Info Torznab search in GloDLS => Found 15 releases (from cache)
Nov 15 11:12:02 vdr mono[2403]: [Warn] Torznab: Indexer Jackett - GloDLS rss sync didn't cover the period between 11/15/2022 09:56:34 and 11/15/2022 09:56:34 UTC. Search ma>
Nov 15 11:12:06 vdr mono[2403]: [Info] DownloadDecisionMaker: Processing 128 releases
Nov 15 11:12:06 vdr mono[2403]: [Info] RssSyncService: RSS Sync Completed. Reports found: 128, Reports grabbed: 0
Nov 15 11:13:17 vdr jackett[448]: 11-15 11:13:17 Info Torznab search in GloDLS => Found 15 releases (from cache)
Nov 15 11:13:17 vdr jackett[448]: 11-15 11:13:17 Info Torznab search in xxx.xxx => Found 13 releases (from cache)
Nov 15 11:14:40 vdr systemd-logind[468]: The system will power off now!
Nov 15 11:14:40 vdr systemd-logind[468]: System is powering down.
Nov 15 05:50:02 vdr postfix/pickup[103810]: 101845B75107: uid=169 from=<transmission>
Nov 15 05:50:02 vdr postfix/cleanup[106169]: 101845B75107: message-id=<20221115045002.101845B75107@vdr.localdomain>
Nov 15 05:50:02 vdr postfix/qmgr[23724]: 101845B75107: from=<transmission@vdr.localdomain>, size=1202, nrcpt=1 (queue active)
Nov 15 05:50:02 vdr postfix/local[106174]: 101845B75107: to=<transmission@vdr.localdomain>, orig_to=<transmission>, relay=local, delay=0.04, delays=0.03/0.01/0/0, dsn=2.0.0>
Nov 15 05:50:02 vdr postfix/qmgr[23724]: 101845B75107: removed
Nov 15 05:50:31 vdr jackett[477]: 11-15 05:50:31 Info Torznab search in AggregateSearch => Found 381 releases
Nov 15 05:50:31 vdr Lidarr[18715]: [Info] DownloadDecisionMaker: Processing 572 releases
Nov 15 05:50:31 vdr Lidarr[18715]: SQLite error (5): database is locked in "PRAGMA cache_size=-10000"
Nov 15 05:50:31 vdr Lidarr[18715]: [Info] RssSyncService: RSS Sync Completed. Reports found: 572, Reports grabbed: 0
Nov 15 05:50:55 vdr systemd-logind[517]: The system will power off now!
Nov 15 05:50:55 vdr systemd-logind[517]: System is powering down.

The server is in the cellar, no kids could reach them and pressing the power button (their sleeping anyway at that time - at least I hope so).

I am a bit puzzled. Could it be the kernel update? Could it be any of the services or any update thereof? Sometimes there is still an open shell dangling before the shutdown, sometimes nobody is logged in. Hence, it must be a service, I assume.

Maybe someone has an idea how to track down this issue further.

Thanks in advance!

Last edited by marlemion (2022-11-18 16:04:23)

Offline

#2 2022-11-16 12:28:11

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [SOLVED] Random shutdowns preferably at night

What does your /etc/systemd/logind.conf look like?
What if you alter all the default shutdowns (and hibernations) to "ignore" (and restart logind)?
Does it happen if you cut the sytem from the network (pull the ethernet, I assume, plug) "over night"?

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
systemctl list-timers

Offline

#3 2022-11-16 13:21:26

marlemion
Member
Registered: 2022-11-16
Posts: 6

Re: [SOLVED] Random shutdowns preferably at night

Thanks for your answer. I have thought about unplugging the network cable for testing. But I should 'remove' bit by bit and see what happens.

vdr # systemctl list-timers
NEXT                        LEFT                LAST                        PASSED            UNIT                             ACTIVATES
Wed 2022-11-16 15:00:39 CET 46min left          Wed 2022-11-16 14:00:14 CET 14min ago         withings-garmin-v2.timer         withings-garmin-v2.service
Thu 2022-11-17 00:00:00 CET 9h left             Wed 2022-11-16 00:00:01 CET 14h ago           shadow.timer                     shadow.service
Thu 2022-11-17 00:32:57 CET 10h left            Wed 2022-11-16 12:54:19 CET 1h 20min ago      certbot.timer                    certbot.service
Thu 2022-11-17 06:42:23 CET 16h left            Wed 2022-11-16 09:47:39 CET 4h 26min ago      man-db.timer                     man-db.service
Thu 2022-11-17 06:51:07 CET 16h left            Wed 2022-11-16 06:51:07 CET 7h ago            systemd-tmpfiles-clean.timer     systemd-tmpfiles-clean.service
Mon 2022-11-21 00:26:35 CET 4 days left         Mon 2022-11-14 00:51:59 CET 2 days ago        fstrim.timer                     fstrim.service
Sat 2022-11-26 11:39:13 CET 1 week 2 days left  Mon 2022-11-14 01:40:33 CET 2 days ago        archlinux-keyring-wkd-sync.timer archlinux-keyring-wkd-sync.service
Mon 2022-12-05 14:45:48 CET 2 weeks 5 days left Fri 2022-11-04 00:01:20 CET 1 week 5 days ago btrfs-scrub@-.timer              btrfs-scrub@-.service

8 timers listed.
Pass --all to see loaded but inactive timers, too.
vdr # find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
acct.service                             | multi-user.target.wants
avahi-daemon.service                     | multi-user.target.wants
avahi-daemon.socket                      | sockets.target.wants
bluetooth.service                        | bluetooth.target.wants
btrfs-scrub@-.timer                      | multi-user.target.wants
certbot.timer                            | timers.target.wants
cronie.service                           | multi-user.target.wants
cups-browsed.service                     | multi-user.target.wants
datafari.service                         | multi-user.target.wants
dbus-org.bluez.service                   | system
dbus-org.freedesktop.Avahi.service       | system
dbus-org.freedesktop.resolve1.service    | system
dbus-org.freedesktop.timesync1.service   | system
docker.service                           | multi-user.target.wants
fstrim.timer                             | multi-user.target.wants
fstrim.timer                             | timers.target.wants
getty@tty1.service                       | getty.target.wants
httpd.service                            | multi-user.target.wants
jackett.service                          | multi-user.target.wants
lidarr.service                           | multi-user.target.wants
lm_sensors.service                       | multi-user.target.wants
mariadb.service                          | multi-user.target.wants
mssql-server.service                     | multi-user.target.wants
nfs-server.service                       | multi-user.target.wants
nmb.service                              | multi-user.target.wants
ntpdate.service                          | multi-user.target.wants
ocrmypdf-web.service                     | multi-user.target.wants
openvpn-client@pia.service               | multi-user.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire.socket                          | sockets.target.wants
portainer.service                        | multi-user.target.wants
postfix.service                          | multi-user.target.wants
pulseaudio.socket                        | sockets.target.wants
radarr.service                           | multi-user.target.wants
remote-fs.target                         | multi-user.target.wants
rngd.service                             | sysinit.target.wants
sabnzbd.service                          | multi-user.target.wants
saned.socket                             | sockets.target.wants
smb.service                              | multi-user.target.wants
sonarr.service                           | multi-user.target.wants
sshd.service                             | multi-user.target.wants
syncthing@xxx.service                   | multi-user.target.wants
systemd-networkd.service                 | multi-user.target.wants
systemd-networkd.socket                  | sockets.target.wants
systemd-resolved.service                 | multi-user.target.wants
systemd-timesyncd.service                | sysinit.target.wants
transmission.service                     | multi-user.target.wants
vdr.service                              | multi-user.target.wants
withings-garmin-v2.timer                 | timers.target.wants

vdr # cat /etc/systemd/logind.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the logind.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/logind.conf' to display the full config.
#
# See logind.conf(5) for details.

[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#UserStopDelaySec=10
#HandlePowerKey=poweroff
#HandlePowerKeyLongPress=ignore
#HandleRebootKey=reboot
#HandleRebootKeyLongPress=poweroff
#HandleSuspendKey=suspend
#HandleSuspendKeyLongPress=hibernate
#HandleHibernateKey=hibernate
#HandleHibernateKeyLongPress=ignore
#HandleLidSwitch=suspend
#HandleLidSwitchExternalPower=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#RebootKeyIgnoreInhibited=no
#HoldoffTimeoutSec=30s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RuntimeDirectoryInodesMax=
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192
#StopIdleSessionSec=infinity

I will try unplugging the network cable tonight. Then I will alter the logind.conf, unless you find something suspicious in the data above.

Offline

#4 2022-11-16 13:35:54

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [SOLVED] Random shutdowns preferably at night

vdr.service yells at me:

http://tvdr.de/features.htm wrote:

Automatic shutdown/wakeup (with certain mainboards)

The other non-common stuff (torrent services) doesn't look very prone to shut down the system.

Offline

#5 2022-11-16 13:49:49

marlemion
Member
Registered: 2022-11-16
Posts: 6

Re: [SOLVED] Random shutdowns preferably at night

seth wrote:

vdr.service yells at me:

http://tvdr.de/features.htm wrote:

Automatic shutdown/wakeup (with certain mainboards)

The other non-common stuff (torrent services) doesn't look very prone to shut down the system.

Good catch. See here: Auto Shutdown verhindern

The values I had in setup.conf were higher than zero. But it is not clear to me why it did not happen before. Nevertheless, I have changed them to zero and will not pull the network cable tonight.

Let's see, I will report. Thank you!

Update:

See here: Stromspareinstellungen

There they say that the vnsi plugin resets the values indicated above at each connection. I use the vnsi plugin to connect our kodi boxes to the server. One of these boxes was running 24/7 and has been replaced by something else during the hardware exchange. The something else is not anymore running 24/7 as ist hosts kodi exlusively only (the box used to have some other services running to balance load of the servers). Hence, since a few days these values have not been resetted at night and vdr should have attempted a shutdown. This also explains the varying shutdown times as they have been adjusted by the time the TV box has been shutdown.

That actually has to be it. We will see.

Last edited by marlemion (2022-11-16 13:59:47)

Offline

#6 2022-11-18 16:03:47

marlemion
Member
Registered: 2022-11-16
Posts: 6

Re: [SOLVED] Random shutdowns preferably at night

Just to let you know that the server did not shut down anymore. Thanks for your help. Marking this as solved.

Offline

Board footer

Powered by FluxBB