Arch Linux

traches

Member

Registered: 2019-03-27

Posts: 12

[solved] Help wrangling systemd services for backup jobs

Hi! I've got a bit of a tangled mess of dependencies and conditions that I'm having a hard time sorting. I have a couple specific questions, but I'd also like a sanity check to make sure I'm not doing things the hard way. Here's the overview:

* I have a desktop and a "server" (laptop stuffed in a cabinet).
* My backup scheme is a restic powered triangle between these 2 computers (over ssh) and backblaze b2.
* These backup jobs are handled by systemd services and timers.
* The desktop is power hungry and electricity is expensive, so I'd like it to suspend when idle. (This is the biggest source of complication).
* Idle detection and auto-suspend is handled by swayidle, which is run as a systemd user level service. I'm basically set up according to this guide: https://github.com/swaywm/sway/wiki/Systemd-integration
* These backups take exceedingly variable amounts of time. Sometimes minutes, sometimes days, it all depends on how much has changed since the last one.

My backups all work just fine, the issues are coming from trying to automate them. I'm sure that waking the desktop can be accomplished with WakeSystem= in the local file, and Wake on Lan from the server, but how do I handle stopping and starting swayidle appropriately? I need to stop it whenever either a local backup is running, or a 'server' backup is running, but I'm not sure how to accomplish this. Alternatively, will swayidle work if run as a system level service?

I found 'ConditionPathExists' in the systemd unit man pages; I could have all my various backup jobs touch a file in some directory and remove it when finished to prevent the idle service from starting when a job is in progress. This doesn't handle starting and stopping it though... I found the 'path' unit which almost looks right, but it seems focused on running services when files are present, not absent, and I don't see how to invert that behavior. I could just run a timer every minute or so, but seems like a hack. I'd like it to be event-driven, if possible. Conflicts= is almost what I want, but it doesn't work between system level and user level services.

So yeah, I nearly have a working system here. Systemd was meant for this kind of dependency management, wasn't it? What am I missing?

Thanks!

Last edited by traches (2019-10-24 12:37:53)

loqs

Member

Registered: 2014-03-06

Posts: 17,321

Re: [solved] Help wrangling systemd services for backup jobs

See man 1 systemd-inhibit I do not know of swayidle supports it.

firecat53

Member

From: Lake Stevens, WA, USA

Registered: 2007-05-14

Posts: 1,542

Website

Re: [solved] Help wrangling systemd services for backup jobs

Some ideas that may or may not help:

1. You can try running swayidle as a system service but using the 'User=' directive in the [Service] section. That's really the only way I know of to make a 'user' service aware of system services like suspend/power.

2. For dependency ordering, I've found creating .targets is useful. I'll post the files instead of trying to explain it! Basically, restic requires a daily backup followed by a forget to remove old snapshots. Then a weekly check is done followed by a prune, but those can't run until the backup and forget jobs are complete.

/etc/systemd/system/restic_backup.target:

[Unit]
Description=Restic backup target
Before=restic_prune.target
StopWhenUnneeded=true

/etc/systemd/system/restic_backup.target.wants/

  restic_forget.service (symlink)
  restic_homeserver.service (symlink)

/etc/systemd/system/restic_prune.target:

[Unit]
Description=Restic backup target
After=restic_backup.target
StopWhenUnneeded=true

/etc/systemd/system/restic_prune.target.wants/

  restic_check.service (symlink)
  restic_prune.service (symlink)

/etc/systemd/system/restic_check.service:

[Unit]
Description=Check restic backups
After=restic_prune.service

[Service]
Type=oneshot
EnvironmentFile=/root/.restic_backup_env                     
ExecStart=/usr/local/bin/restic check

[Install]
WantedBy=restic_prune.target

/etc/systemd/system/restic_forget.service:

[Unit]
Description=Forget (delete snapshots) Restic Backups         
After=restic_homeserver.service

[Service]
Type=oneshot
EnvironmentFile=/root/.restic_backup_env                     
ExecStart=/usr/local/bin/restic forget --keep-monthly 6 --keep-weekly 12 --keep-daily 30

[Install]
WantedBy=restic_backup.target

/etc/systemd/system/restic_homeserver.service:

[Unit]
Description=Restic Backup to B2
Before=restic_forget.service
RequiresMountsFor=/home /mnt/media                           

[Service]
Type=oneshot
EnvironmentFile=/root/.restic_backup_env
ExecStart=/usr/local/bin/restic backup /home/ /srv /var/backups /mnt/media

[Install]
WantedBy=restic_backup.target

/etc/systemd/system/restic_prune.service:

[Unit]
Description=Prune Restic Backups
Before=restic_check.service
After=restic_forget.service

[Service]
Type=oneshot
EnvironmentFile=/root/.restic_backup_env                     
ExecStart=/usr/local/bin/restic prune

[Install]
WantedBy=restic_prune.target

---

Github

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [solved] Help wrangling systemd services for backup jobs

Thanks for the answers, and sorry for being slow to reply!

See man 1 systemd-inhibit I do not know of swayidle supports it.

This is almost exactly what I want, but not quite. systemd-inhibit can either delay a suspend operation or block it completely, and while delaying would sort of work, I can't find a way to cancel a delayed suspend operation once started. That means it'll suspend when the backup finishes, whether I'm using the computer or not!

Some ideas that may or may not help:

1. You can try running swayidle as a system service but using the 'User=' directive in the [Service] section. That's really the only way I know of to make a 'user' service aware of system services like suspend/power.

2. For dependency ordering, I've found creating .targets is useful. I'll post the files instead of trying to explain it! Basically, restic requires a daily backup followed by a forget to remove old snapshots. Then a weekly check is done followed by a prune, but those can't run until the backup and forget jobs are complete.

Your backup setup looks really similar to mine! I like the target based approach; I've just created a master 'backup' service, with all the dependencies between services declared explicitly. Your solution looks cleaner; I'll probably take some parts of it.

Regarding the suspend problem, inspiration struck while I was riding the train. I haven't tried to write this yet, but I think it'll work:

I want to create 'lock' files and 'request' files in some arbitrary directory (like /tmp/idle-manager/{locks,requests}/). I'll monitor these directories with a systemd path unit, which will run a script whenever that directory changes. This script will suspend the computer, only if there is one or more request files and no lock files.

Then, I can have all my various backup services touch and remove lock files in that directory as they start and finish. Swayidle is really simple, it just runs arbitrary commands after a timeout (and also on resume). Instead of suspending the computer directly, I'll have it manage a 'request' file. This should get the behavior I want - it'll suspend when truly idle, and not when in use or when a backup is in progress.

I'll report back with results!

traches

Member

Registered: 2019-03-27

Posts: 12

Re: [solved] Help wrangling systemd services for backup jobs

Back with an update as promised.

I didn't quite do it that way, but I think this way is simpler and it works like a charm. I'm actually pretty proud of this thing!

I wrote a little script called sus-man (suspend manager), which has 4 commands: lock, unlock, suspend, and resume, each of which create or remove a lock or a suspend request as appropriate. Each time it's called, after performing the command it call 'systemctl suspend' if and only if there are any suspend requests but no locks. Then, I can have swayidle call sus-mon suspend/resume as appropriate, and all my backup scripts can call sus-mon lock (with a name) when they start and sus-mon unlock (with the same name) when they finish. Locks and requests are just files in particular folders in the tmp directory, so they won't persist across reboots.

The cool part is I can use this for other stuff too. For example I can modify the sshd@ service to create and remove a lock, so it'll never suspend when an SSH session is connected. This makes it easy for my server to 'wake on lan' the desktop, and then once it connects over SSH to perform the backup it won't suspend again until it disconnects.

Here's the script, for anyone curious:

#! /bin/bash
#
# The purpose of this script is to handle intelligent auto-suspend behavior. It
# allows long-running jobs (such as backups and SSH sessions) to prevent
# auto-suspend, but only until they are finished.
#
# It works by allowing the creation and deletion of locks, which prevent the
# computer from auto-suspending, and requests, which will trigger a suspend once
# there are no locks present. To create a lock, call sus-man lock (some name). To
# remove that lock (make sure you do!), call sus-man unlock (same name).
# Suspend requests work the same way, with the suspend & resume arguments.

tmp_dir=/tmp/sys-man
lock_dir=$tmp_dir/locks
request_dir=$tmp_dir/requests

lock() {
  touch $lock_dir/$1
  check_state
}

unlock() {
  rm -f $lock_dir/$1
  check_state
}

# Can't name it suspend, sadly.
set_suspend() {
  touch $request_dir/$1
  check_state
}

resume() {
  rm -f $request_dir/$1
  check_state
}

list() {
  echo "Locks:"
  echo $(ls -A $lock_dir)
  echo "Requests:"
  echo $(ls -A $request_dir)
}

check_state() {
  if [ "$( ls -A $request_dir )" ] && \
    ! [ "$( ls -A $lock_dir )" ]
    then
      echo "sus-man suspend triggered."
       systemctl suspend
  fi
}

show_help() {
  echo 'Usage:'
  echo 'sus-man (operation) (name)'
  echo ''
  echo 'operation: lock, unlock, suspend, or resume.'
  echo 'name:      a name for a given lock or request.'
  echo ''
  echo 'Call with no arguments for a list of current locks and requests.'
}

mkdir -p {$lock_dir,$request_dir}

case $1 in
  'lock') lock $2;;
  'unlock') unlock $2;;
  'suspend') set_suspend $2;;
  'resume') resume $2;;
  '') list ;;
  *) show_help ;;
esac

Here's an example backup script:

#!/bin/bash
# Back up to cloudbucket

/usr/local/bin/sus-man lock cloudbucket-backup

restic backup                                                     \
  --repo sftp:robert@cloudbucket:/bucket/local_backups/charybdis/ \
  --files-from /etc/backup_file_list                              \
  --exclude-file /etc/backup_exclude_list                         \
  --password-file /root/backup_password_file                      \
  --cache-dir /var/cache/restic/                                  \
  --one-file-system                                               \
  --exclude-caches                                                \
  --verbose                                                       \

/usr/local/bin/sus-man unlock cloudbucket-backup

I should modify this so it return restic's exit status instead of sus-man's, but other than that I think this solution should do exactly what I need. Hope this helps someone else, and thanks for all the advice!