You are not logged in.

#1 2023-08-04 16:34:06

grmat
Member
Registered: 2017-05-08
Posts: 3

systemd/user backup routine

Hi,

Setup:

- server1 with git repos of multiple users
- server2 for an additional backup of the git repos

server1 is running permanently and worked with, basically hosting the upstream repos.

What I want to achieve is server2 booting up, fetching the configured repos and shutting down again when it's all done. I thought that would be trivial to do with systemd but I haven't get it done yet.

I can automatically start server2 periodically, so that's out of scope.
I set up a backup@.service template that fetches the given repos for the corresponding user (systemctl --user). My idea was to set up a timed shutdown that depends on the enabled backup services, so it will shutdown after a given uptime, but not before all these deps are met.

I think my primary issue is that user services can't depend on system services and the other way round, therefore I can't utilise stuff like the network-online.target or shutdown.target (even though the latter exists for the user instance as well).

Are there any suggestions to get this task done?
Thanks!

Offline

#2 2023-08-05 03:03:57

ectospasm
Member
Registered: 2015-08-28
Posts: 273

Re: systemd/user backup routine

So each user has a named user account on both systems, and you're having each of those users on server2 download their own repos?  Can you provide an example user service unit you're trying to use?

Do you really need all those user accounts on server2?  If server2 will be powered off most of the time, just wake up to do the transfers, why not have a system service (as root) fetch everything.  You can add server2's root SSH public key in each server1 users' .ssh/authorized_keys, and place all the dependencies (sockets, targets, etc.) in the system unit file.  You can even script it so the server2 system service sets the ownership and permissions correctly for the fetched repos, if that's important to you. 

You could even restrict these root public keys to only be able to do the transfers (with the command syntax in server1:~user/.ssh/authorized_keys), so server2 can only do the fetch, nothing else.

If you really wanted to get fancy you could store the server2 root private SSH passphrase in the fetch service unit with systemd-creds, but this would be better if server2 had a TPM chip in it.

Offline

#3 2023-08-05 05:32:54

grmat
Member
Registered: 2017-05-08
Posts: 3

Re: systemd/user backup routine

Thanks for your answer!

ectospasm wrote:

So each user has a named user account on both systems, and you're having each of those users on server2 download their own repos?

Exactly. The reason I didn't want root on server2 doing all that is that it could work as a drop-in replacement for server1 in case of a hardware issue on server1. It's not much work to set it up when all the data is already there in their $HOMEs, but it's "work" after all. Your remarks sound reasonable, though - I'll have to rethink it.


Can you provide an example user service unit you're trying to use?

As  I don't have it up and running, I've just tried some stuff, I can (only) provide a file with some commented-out states of my failed attempts:

[Unit]
Description=Specific git repo backup routine
#Upholds=exit.target
#DefaultDependencies=no
#Before=shutdown.target
#Before=exit.target
#After=network-online.target
#Requires=network-online.target
#After=systemd-networkd-wait-online.service
#Requires=systemd-networkd-wait-online.service

[Service]
#Type=simple
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/git -C %h/repos/%i fetch
RestartSec=60
Restart=on-failure

[Install]
#WantedBy=default.target
#WantedBy=exit.target
WantedBy=shutdown.target

Not sure about everything anymore, though, and I haven't documented everything. shutdown.target is available for user services but I haven't got the service running as I wanted, as described in the first post, neither with active lingering.

Last edited by grmat (2023-08-05 05:35:48)

Offline

#4 2023-08-05 13:55:15

ectospasm
Member
Registered: 2015-08-28
Posts: 273

Re: systemd/user backup routine

So how are you triggering these instantiated template user services?  I'd set a timer with OnBootSec=5m (possibly with a RandomizedDelaySec= so they all don't run at exactly the same time) for each, and have an ExecStartPost= in the user service template to write to some log (in /var/log/, or even better, /tmp, so it won't survive a reboot) that writes when each instantiated user service completes, and have another timer that reads this log and initiates the shutdown once all of the users have fetched their repos.

Or you could have the completion detection script read the journal, and only when all the user services have completed, only then trigger the shutdown.  And actually, you may be able to make this a system template, just add the User=%i to the [Service] section so it gets executed as that user.  You'd have to put the path /home/%i instead of %h since the latter would be /root for a system service.

Offline

#5 2023-08-05 14:02:24

grmat
Member
Registered: 2017-05-08
Posts: 3

Re: systemd/user backup routine

ectospasm wrote:

So how are you triggering these instantiated template user services?

Right now I just set them up for the default.target, then shutdown after a given amount of time. But the the initial idea was to set them as a dependency for e.g. shutdown.target to have them pulled in before powering off, but I didn't get this up, which is why we're here.

And actually, you may be able to make this a system template, just add the User=%i to the [Service] section so it gets executed as that user.

That's an excellent hint, I thought of setting User= in a system service but didn't think of parametrizing that value as well. I'll try this, I think it might do it.

The idea with the delays is also helpful, thanks a lot.

Offline

#6 2023-08-05 14:24:26

ectospasm
Member
Registered: 2015-08-28
Posts: 273

Re: systemd/user backup routine

If you put them as dependencies of the shutdown target, you may not have to specify any dependencies in the template unit.  You could even make it if any of the instances fail, shutdown will fail and you can send yourself an alert.

Offline

#7 2023-08-05 14:29:38

ectospasm
Member
Registered: 2015-08-28
Posts: 273

Re: systemd/user backup routine

So, unit files have the OnFailure= directive (in the [Unit] section), which you can run your notification command(s) to alert you if the instance fails.

Last edited by ectospasm (2023-08-05 15:19:16)

Offline

Board footer

Powered by FluxBB