You are not logged in.
- Topics: Active | Unanswered
#1 2022-11-16 08:08:21
- marlemion
- Member
- Registered: 2022-11-16
- Posts: 6
[SOLVED] Random shutdowns preferably at night
Dear all,
I have several arch installs running, in particular one of them being the main server of our house hosting quite a few services such as nextcloud docker, http server, NFS and SMB file servers, Lidarr, Radarr, Sonarr, MySQL DB, vdr etc.
This setup has been installed many years ago and has been migrated from a few hardwares already. A few days back I encountered a spontaneous shutdown at night (3am CET). As I wanted to change the hardware anyway I took the opportunity and migrated the system from the original hardware (ASRock BeeBox N3160 w. 8GB RAM) to a new system (Intel NUC8i3BEH w. 40GB RAM). The remaining hardware was the same, i.e. 2TB internal, 2*2TB USB, two DVB-S USB Receiver. Still, the shutdowns persist.
last -x | egrep -E 'shutdown|reboot' | head -n 40
reboot system boot 6.0.8-arch1-1 Wed Nov 16 06:36 still running
shutdown system down 6.0.8-arch1-1 Wed Nov 16 04:09 - 06:36 (02:26)
reboot system boot 6.0.8-arch1-1 Tue Nov 15 18:05 - 04:09 (10:03)
shutdown system down 6.0.8-arch1-1 Tue Nov 15 11:15 - 18:05 (06:50)
reboot system boot 6.0.8-arch1-1 Tue Nov 15 06:14 - 11:15 (05:00)
shutdown system down 6.0.8-arch1-1 Tue Nov 15 05:51 - 06:14 (00:23)
reboot system boot 6.0.8-arch1-1 Mon Nov 14 09:27 - 05:51 (20:23)
shutdown system down 6.0.6-arch1-1 Mon Nov 14 09:26 - 09:27 (00:00)
reboot system boot 6.0.6-arch1-1 Mon Nov 14 06:18 - 09:26 (03:08)
shutdown system down 6.0.6-arch1-1 Mon Nov 14 03:36 - 06:18 (02:41)
LOTS OF INTENDED REBOOTS BEFORE DUE TO MIGRATION
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:25 - 03:36 (14:11)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 13:22 - 13:25 (00:02)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:20 - 13:22 (00:01)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 13:18 - 13:20 (00:02)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:16 - 13:18 (00:01)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 13:12 - 13:16 (00:03)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:11 - 13:12 (00:01)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 13:10 - 13:11 (00:00)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:08 - 13:10 (00:02)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 13:07 - 13:08 (00:00)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 13:05 - 13:07 (00:01)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 12:57 - 13:07 (00:09)
shutdown system down 6.0.6-arch1-1 Sun Nov 13 12:56 - 12:57 (00:01)
reboot system boot 6.0.6-arch1-1 Sun Nov 13 12:28 - 12:56 (00:27)
NEW HARDWARE
shutdown system down 6.0.6-arch1-1 Sat Nov 12 18:43 - 12:28 (17:45)
reboot system boot 6.0.6-arch1-1 Sat Nov 12 08:03 - 18:43 (10:39)
shutdown system down 6.0.6-arch1-1 Sat Nov 12 04:49 - 08:03 (03:13)
reboot system boot 6.0.6-arch1-1 Fri Nov 11 18:55 - 04:49 (09:53)
shutdown system down 6.0.6-arch1-1 Fri Nov 11 03:31 - 18:55 (15:23)
reboot system boot 6.0.6-arch1-1 Mon Oct 31 18:28 - 03:31 (10+09:03)
NEW KERNEL
shutdown system down 5.18.16-arch1-1 Mon Oct 31 18:27 - 18:28 (00:00)As you can see, I have performed an update before the first reboot at 3:31am Nov 11 occurred (new kernel). From there on I had constant shutdowns every night in between 3 and 6 am with one exception: Nov 15 at 11:15am. It is completely weird. I have disabled all cronjobs - no effect. The logs are not helpful. Excerpts:
Nov 16 04:01:01 vdr CROND[32828]: (root) CMD (run-parts /etc/cron.hourly)
Nov 16 04:01:01 vdr CROND[32827]: (root) CMDEND (run-parts /etc/cron.hourly)
Nov 16 04:03:07 vdr Radarr[2511]: SQLite error (5): database is locked in "PRAGMA cache_size=-20000"
Nov 16 04:03:32 vdr vdr[478]: [478] ERROR: no OSD provider available - using dummy OSD!
Nov 16 04:08:33 vdr systemd-logind[471]: The system will power off now!
Nov 16 04:08:33 vdr systemd-logind[471]: System is powering down.Nov 15 11:12:02 vdr mono[2403]: [Info] RssSyncService: Starting RSS Sync
Nov 15 11:12:02 vdr jackett[448]: 11-15 11:12:02 Info Torznab search in xxx.xxx => Found 13 releases (from cache)
Nov 15 11:12:02 vdr jackett[448]: 11-15 11:12:02 Info Torznab search in GloDLS => Found 15 releases (from cache)
Nov 15 11:12:02 vdr mono[2403]: [Warn] Torznab: Indexer Jackett - GloDLS rss sync didn't cover the period between 11/15/2022 09:56:34 and 11/15/2022 09:56:34 UTC. Search ma>
Nov 15 11:12:06 vdr mono[2403]: [Info] DownloadDecisionMaker: Processing 128 releases
Nov 15 11:12:06 vdr mono[2403]: [Info] RssSyncService: RSS Sync Completed. Reports found: 128, Reports grabbed: 0
Nov 15 11:13:17 vdr jackett[448]: 11-15 11:13:17 Info Torznab search in GloDLS => Found 15 releases (from cache)
Nov 15 11:13:17 vdr jackett[448]: 11-15 11:13:17 Info Torznab search in xxx.xxx => Found 13 releases (from cache)
Nov 15 11:14:40 vdr systemd-logind[468]: The system will power off now!
Nov 15 11:14:40 vdr systemd-logind[468]: System is powering down.Nov 15 05:50:02 vdr postfix/pickup[103810]: 101845B75107: uid=169 from=<transmission>
Nov 15 05:50:02 vdr postfix/cleanup[106169]: 101845B75107: message-id=<20221115045002.101845B75107@vdr.localdomain>
Nov 15 05:50:02 vdr postfix/qmgr[23724]: 101845B75107: from=<transmission@vdr.localdomain>, size=1202, nrcpt=1 (queue active)
Nov 15 05:50:02 vdr postfix/local[106174]: 101845B75107: to=<transmission@vdr.localdomain>, orig_to=<transmission>, relay=local, delay=0.04, delays=0.03/0.01/0/0, dsn=2.0.0>
Nov 15 05:50:02 vdr postfix/qmgr[23724]: 101845B75107: removed
Nov 15 05:50:31 vdr jackett[477]: 11-15 05:50:31 Info Torznab search in AggregateSearch => Found 381 releases
Nov 15 05:50:31 vdr Lidarr[18715]: [Info] DownloadDecisionMaker: Processing 572 releases
Nov 15 05:50:31 vdr Lidarr[18715]: SQLite error (5): database is locked in "PRAGMA cache_size=-10000"
Nov 15 05:50:31 vdr Lidarr[18715]: [Info] RssSyncService: RSS Sync Completed. Reports found: 572, Reports grabbed: 0
Nov 15 05:50:55 vdr systemd-logind[517]: The system will power off now!
Nov 15 05:50:55 vdr systemd-logind[517]: System is powering down.The server is in the cellar, no kids could reach them and pressing the power button (their sleeping anyway at that time - at least I hope so).
I am a bit puzzled. Could it be the kernel update? Could it be any of the services or any update thereof? Sometimes there is still an open shell dangling before the shutdown, sometimes nobody is logged in. Hence, it must be a service, I assume.
Maybe someone has an idea how to track down this issue further.
Thanks in advance!
Last edited by marlemion (2022-11-18 16:04:23)
Offline
#2 2022-11-16 12:28:11
- seth
- Member
- Registered: 2012-09-03
- Posts: 59,897
Re: [SOLVED] Random shutdowns preferably at night
What does your /etc/systemd/logind.conf look like?
What if you alter all the default shutdowns (and hibernations) to "ignore" (and restart logind)?
Does it happen if you cut the sytem from the network (pull the ethernet, I assume, plug) "over night"?
find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
systemctl list-timersOffline
#3 2022-11-16 13:21:26
- marlemion
- Member
- Registered: 2022-11-16
- Posts: 6
Re: [SOLVED] Random shutdowns preferably at night
Thanks for your answer. I have thought about unplugging the network cable for testing. But I should 'remove' bit by bit and see what happens.
vdr # systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2022-11-16 15:00:39 CET 46min left Wed 2022-11-16 14:00:14 CET 14min ago withings-garmin-v2.timer withings-garmin-v2.service
Thu 2022-11-17 00:00:00 CET 9h left Wed 2022-11-16 00:00:01 CET 14h ago shadow.timer shadow.service
Thu 2022-11-17 00:32:57 CET 10h left Wed 2022-11-16 12:54:19 CET 1h 20min ago certbot.timer certbot.service
Thu 2022-11-17 06:42:23 CET 16h left Wed 2022-11-16 09:47:39 CET 4h 26min ago man-db.timer man-db.service
Thu 2022-11-17 06:51:07 CET 16h left Wed 2022-11-16 06:51:07 CET 7h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2022-11-21 00:26:35 CET 4 days left Mon 2022-11-14 00:51:59 CET 2 days ago fstrim.timer fstrim.service
Sat 2022-11-26 11:39:13 CET 1 week 2 days left Mon 2022-11-14 01:40:33 CET 2 days ago archlinux-keyring-wkd-sync.timer archlinux-keyring-wkd-sync.service
Mon 2022-12-05 14:45:48 CET 2 weeks 5 days left Fri 2022-11-04 00:01:20 CET 1 week 5 days ago btrfs-scrub@-.timer btrfs-scrub@-.service
8 timers listed.
Pass --all to see loaded but inactive timers, too.
vdr # find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
acct.service | multi-user.target.wants
avahi-daemon.service | multi-user.target.wants
avahi-daemon.socket | sockets.target.wants
bluetooth.service | bluetooth.target.wants
btrfs-scrub@-.timer | multi-user.target.wants
certbot.timer | timers.target.wants
cronie.service | multi-user.target.wants
cups-browsed.service | multi-user.target.wants
datafari.service | multi-user.target.wants
dbus-org.bluez.service | system
dbus-org.freedesktop.Avahi.service | system
dbus-org.freedesktop.resolve1.service | system
dbus-org.freedesktop.timesync1.service | system
docker.service | multi-user.target.wants
fstrim.timer | multi-user.target.wants
fstrim.timer | timers.target.wants
getty@tty1.service | getty.target.wants
httpd.service | multi-user.target.wants
jackett.service | multi-user.target.wants
lidarr.service | multi-user.target.wants
lm_sensors.service | multi-user.target.wants
mariadb.service | multi-user.target.wants
mssql-server.service | multi-user.target.wants
nfs-server.service | multi-user.target.wants
nmb.service | multi-user.target.wants
ntpdate.service | multi-user.target.wants
ocrmypdf-web.service | multi-user.target.wants
openvpn-client@pia.service | multi-user.target.wants
p11-kit-server.socket | sockets.target.wants
pipewire.socket | sockets.target.wants
portainer.service | multi-user.target.wants
postfix.service | multi-user.target.wants
pulseaudio.socket | sockets.target.wants
radarr.service | multi-user.target.wants
remote-fs.target | multi-user.target.wants
rngd.service | sysinit.target.wants
sabnzbd.service | multi-user.target.wants
saned.socket | sockets.target.wants
smb.service | multi-user.target.wants
sonarr.service | multi-user.target.wants
sshd.service | multi-user.target.wants
syncthing@xxx.service | multi-user.target.wants
systemd-networkd.service | multi-user.target.wants
systemd-networkd.socket | sockets.target.wants
systemd-resolved.service | multi-user.target.wants
systemd-timesyncd.service | sysinit.target.wants
transmission.service | multi-user.target.wants
vdr.service | multi-user.target.wants
withings-garmin-v2.timer | timers.target.wants
vdr # cat /etc/systemd/logind.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it under the
# terms of the GNU Lesser General Public License as published by the Free
# Software Foundation; either version 2.1 of the License, or (at your option)
# any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the logind.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/logind.conf' to display the full config.
#
# See logind.conf(5) for details.
[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#UserStopDelaySec=10
#HandlePowerKey=poweroff
#HandlePowerKeyLongPress=ignore
#HandleRebootKey=reboot
#HandleRebootKeyLongPress=poweroff
#HandleSuspendKey=suspend
#HandleSuspendKeyLongPress=hibernate
#HandleHibernateKey=hibernate
#HandleHibernateKeyLongPress=ignore
#HandleLidSwitch=suspend
#HandleLidSwitchExternalPower=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#RebootKeyIgnoreInhibited=no
#HoldoffTimeoutSec=30s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RuntimeDirectoryInodesMax=
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192
#StopIdleSessionSec=infinityI will try unplugging the network cable tonight. Then I will alter the logind.conf, unless you find something suspicious in the data above.
Offline
#4 2022-11-16 13:35:54
- seth
- Member
- Registered: 2012-09-03
- Posts: 59,897
Re: [SOLVED] Random shutdowns preferably at night
vdr.service yells at me:
Automatic shutdown/wakeup (with certain mainboards)
The other non-common stuff (torrent services) doesn't look very prone to shut down the system.
Offline
#5 2022-11-16 13:49:49
- marlemion
- Member
- Registered: 2022-11-16
- Posts: 6
Re: [SOLVED] Random shutdowns preferably at night
vdr.service yells at me:
http://tvdr.de/features.htm wrote:Automatic shutdown/wakeup (with certain mainboards)
The other non-common stuff (torrent services) doesn't look very prone to shut down the system.
Good catch. See here: Auto Shutdown verhindern
The values I had in setup.conf were higher than zero. But it is not clear to me why it did not happen before. Nevertheless, I have changed them to zero and will not pull the network cable tonight.
Let's see, I will report. Thank you!
Update:
See here: Stromspareinstellungen
There they say that the vnsi plugin resets the values indicated above at each connection. I use the vnsi plugin to connect our kodi boxes to the server. One of these boxes was running 24/7 and has been replaced by something else during the hardware exchange. The something else is not anymore running 24/7 as ist hosts kodi exlusively only (the box used to have some other services running to balance load of the servers). Hence, since a few days these values have not been resetted at night and vdr should have attempted a shutdown. This also explains the varying shutdown times as they have been adjusted by the time the TV box has been shutdown.
That actually has to be it. We will see.
Last edited by marlemion (2022-11-16 13:59:47)
Offline
#6 2022-11-18 16:03:47
- marlemion
- Member
- Registered: 2022-11-16
- Posts: 6
Re: [SOLVED] Random shutdowns preferably at night
Just to let you know that the server did not shut down anymore. Thanks for your help. Marking this as solved.
Offline