Custom shutdown service to backup files to server times out

mindstormer · 2017-03-03 23:43:03

I must have restarted several dozen times via trial-and-error in order to try to get my backup service to work. It simply executes a script that backs up my Firefox bookmarks on shutdown (as is, it works on restart too but that's for debugging purposes) via rsync. The script always takes less than 20 seconds to run and has no issues being executed manually at all. journalctl -u my.service indicates that it successfullly ssh's into my machine, but the service always times out or is stuck indefinitely. A partial file is copied to the server and I noticed that the size of this partial size varies each time. This leads me to suspect that NetworkManager.service or the internet connection is terminated while this script is executing on shutdown. I tried both After=network.target and Requires=NetworkManager.service and some variations of this.

Other information that may be of interest: /home is a separate btrfs subvolume, maybe it gets unmounted during this time.

[Unit]
Description=Backup Firefox bookmarks
After=network.target

[Service]
User=mindstormer
Type=oneshot
ExecStart=/bin/true
ExecStop=/home/mindstomer/scripts/backup.sh
RemainAfterExit=yes
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

Any suggestions or tips are much appreciated, even if it helps in diagnosing the issue.

Last edited by mindstormer (2017-03-03 23:44:49)

berbae · 2017-03-04 10:00:23

I think the problem is you start your script as an 'ExecStop=' option at shutdown time.
It seems everything runs in parallel during the shutdown sequence, without a means to influence it securely (at least I didn't find one).
So I think the most reliable and easy way to do what you want is to modify the service like that:

[Unit]
Description=Backup Firefox bookmarks

[Service]
User=mindstormer
Type=oneshot
ExecStart=/home/mindstomer/scripts/backup.sh

And to use a script which forces the sequence to shutdown your machine:

systemctl start my.service
systemctl poweroff

(poweroff or halt or reboot)
Maybe add a 'sleep 30' between the two commands if necessary.

seth · 2017-03-04 15:30:05

Any reason you're not using /usr/lib/systemd/system-shutdown - just drop a script there, should look like

#!/bin/sh
case $1/$2 in
        pre/*)
        /home/mindstomer/scripts/backup.sh # maybe "sudo -u mindstormer ..."?
        ;;
esac

fayr · 2017-03-19 16:31:25

I got sent here. @berbae, I'd rather see if the service can be fixed rather than relying on a script to poweroff, which adds unnecessary complexity (since I manage multiple machines, powering off a machine should be a simple "systemctl poweroff", for example. From googling, people were able to get it working with a very similar if not the exact same service (example: http://unix.stackexchange.com/questions … -shutdown). The things that do run in parallel that might conflict with the service are all dealt with, aren't they? Via in my case,

[Unit]
Description...
RequiresMountsFor=/home
Requires=network-online.target
After=network-online.target

I have the exact same service and problem as OP other than the above lines to replace under [Unit].

@seth

I've read that people were not able to get /usr/lib/systemd/system-shutdown working for backups because a script executed there is typically far too late in the shutdown (http://unix.stackexchange.com/questions … or-shutdow) process for the script to be able to work. Also, I prefer systemd services as they seem to be more conventional, flexible, and easier to use. I don't have too much scripting experience, admittedly.

Last edited by fayr (2017-03-19 16:35:16)

seth · 2017-03-20 14:36:18

I'm not gonna even ask about your backup strategy when a ro mounted local FS is a problem, or how often you shutdown the system for this to be a reasonable backup strategy, but see https://superuser.com/questions/1016827 … th-systemd or https://ubuntuforums.org/showthread.php?t=2317181 on how to cheat this, but notice that
a) long term processes are not officially supported for ExecStop and
b) backup on shutdown is not a good backup strategy at all

=> backup using a cron job or inotify hooks and inhibit shutdown while it's running...

PS: I'm not sure about the ro-remount condition for /usr/lib/systemd/system-shutdown, but the scripts there (eg. a alink to your backup script ...) should be executed before the system actually seeks the poweroff/reboot target

mindstormer · 2017-03-23 13:31:02

@seth I'm not sure what you mean by ro in the context of the problem or why backing up some data on shutdown is so terrible, but the links you mentioned I've actually based my service off of. I decided to test my service on my desktop machine to try to see if I can get it to work, but turns out that the following service works on the desktop machine but does not work on the laptop for some reason:

[Unit]
Description=Backup Firefox bookmarks
RequiresMountsFor=/home
Requires=network-online.target
After=network-online.target

[Service]
User=mindstormer
Type=oneshot
ExecStart=/bin/true
ExecStop=/home/mindstomer/scripts/backup.sh
RemainAfterExit=yes
StandardOutput=tty

[Install]
WantedBy=multi-user.target

As far as I know, the only difference between my desktop machine and the laptop is that I use systemd-networkd.service for wired internet on the desktop and NetworkManager.service for wireless internet for the laptop (tested only on another network but that shouldn't be relevant). From what I've read, network-online.target should work for NetworkManager.service.

Arch Linux

#1 2017-03-03 23:43:03

Custom shutdown service to backup files to server times out

#2 2017-03-04 10:00:23

Re: Custom shutdown service to backup files to server times out

#3 2017-03-04 15:30:05

Re: Custom shutdown service to backup files to server times out

#4 2017-03-19 16:31:25

Re: Custom shutdown service to backup files to server times out

#5 2017-03-20 14:36:18

Re: Custom shutdown service to backup files to server times out

#6 2017-03-23 13:31:02

Re: Custom shutdown service to backup files to server times out

Board footer