[SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

lamarpavel · 2016-06-07 15:11:53

Hi there,

I have set up duplicity to push data to a server through SSH. The configuration is split between ~/.ssh/config where the host, user and id are defined and a shell script with just the duplicity command.

Invoking said shell script manually works as expected, but when invoked by the following systemd service,

[Unit]
Description=Duplicity backup service for user %I
[Service]
Type=oneshot
User=%i
ExecStart=/home/lamar/.local/bin/backup.sh
StandardOutput=syslog
StandardError=syslog

it fails:

Jun 07 16:52:35 Server backup.sh[32327]: BackendException: ssh connection to <redacted> failed: not a valid EC private key file

So the script is executed, duplicity starts and then throws this error. The unit file is under ~/.config/systemd/user/backup.service and I start it with systemctl --user start backup.service.

Searching the net revealed that the error is thrown by paramiko, the SSH backend used by duplicity. However, I failed to find bug reports or solutions that gave me a hint beyond the fact that my ssh key doen't seem to be found. The id file to be used for this host is defined in ~/.ssh/conf, which works when invoking the script manually. I tried passing the ssh-option to duplicity explicitly, but the result is the same.

This could be a bug with paramiko, but it might very well be related to a lack of knowledge on my part about how systemd invokes user services. Are there common pitfalls that I might be missing?

edit:
I just tried the alternative SSH backend for duplicity "pexpect", which works when invoked manually as well, but fails when invoked by the systemd service. However, the error is different:

Jun 07 17:15:45 Server backup.sh[4379]: Giving up after 5 attempts. AttributeError: 'NoneType' object has no attribute 'encode'

This seems to be a python exception, though I can't tell if it's coming from duplicity or pexpect.

Last edited by lamarpavel (2016-06-08 15:26:45)

ewaller · 2016-06-07 15:18:06

Not sure, but I do not believe that '~' is defined for systemd services. Again, I think that is more of a shell construct. I am not even certain that $HOME is defined, but I could be wrong.
You might look at your script and provide an absolute path to the keys. When you say you run the script manually, you are not sourcing the script, are you? IOW, the script has the executable bit set.

lamarpavel · 2016-06-07 15:23:44

ewaller wrote:

Not sure, but I do not believe that '~' is defined for systemd services. Again, I think that is more of a shell construct. I am not even certain that $HOME is defined, but I could be wrong.
You might look at your script and provide an absolute path to the keys. When you say you run the script manually, you are not sourcing the script, are you? IOW, the script has the executable bit set.

The script is executable and in my PATH, I run it simply with "backup.sh" when doing it manually.

The argument passed to duplicity in the script is using an absolute path:

--ssh-options="-oIdentityFile='/home/lamar/.ssh/id_backup'"

Last edited by lamarpavel (2016-06-07 15:24:30)

ewaller · 2016-06-07 15:31:43

My suggestion would be to update the script to use logger to send debug messages to the journal. That way you can see if the script is being launched, and could give you insight as to its state.

lamarpavel · 2016-06-07 15:47:27

Thanks for the hint, I didn't know about logger and your suggestion helped me find a problem: The SSH_AGENT_PID and SSH_AUTH_SOCK are not set when my script is invoked by the systemd service. I'll go read some manpages to find out how to get access to the env there.

lamarpavel · 2016-06-07 16:16:15

Indeed that was the issue. For the record:

systemctl --user set-environment SSH_AGENT_PID=$SSH_AGENT_PID
systemctl --user set-environment SSH_AUTH_SOCK=$SSH_AUTH_SOCK
systemctl --user daemon-reload && systemctl --user start backup.service

This now works.

One last problem remains: I could add the first two lines to .zprofile or .xinitrc, but according to the manpage this will only affect services started afterwards and if those envvars change later on, the service won't know.
Is there a way to tell the service to somehow import something from the users environment before ExecStart?

firecat53 · 2016-06-07 17:00:31

[Service]
....
Environment=HOME=%h
....

See 'man systemd.unit'

Scott

lamarpavel · 2016-06-07 20:54:23

firecat53 wrote:

[Service]
....
Environment=HOME=%h
....
See 'man systemd.unit'
Scott

Sorry, but I can't figure out how this helps. $HOME already has the correct value, and that line has no effect on the variables that I need. I have tried Environment=SSH_AGENT_PID=$SSH_AGENT_PID but this only sets "$SSH_AGENT_PID" literally as the environment variable and there seems to be no access to my users env.
I also tried to ExecStart=/usr/bin/systemctl --user set-environment SSH_AGENT_PID=$SSH_AGENT_PID, but this doesn't work due to taking effect only for services starting afterwards.

firecat53 · 2016-06-07 23:47:33

Sorry, the HOME= example was an example to point you in the right direction. I should have made that clear

I have a recurring rsync job run from a systemd user unit that needs access to SSH_AUTH_SOCK (I don't need the SSH_AGENT_PID) to acquire the key from ssh-agent. So the line in the unit file is actually pointed at the location of the socket:

Environment=SSH_AUTH_SOCK=/run/user/1000/ssh_auth_sock

If you need other variables imported, this wiki section has an example of using the --import-environment directive in .bash_profile to pass those variables to all 'systemctl --user' instances. There's also some more info in 'man systemd.exec'.

Hope that helped more!
Scott

lamarpavel · 2016-06-08 15:25:42

Alright, with your help I have it working now. To sum it up for anyone with the same issue:

Any process launched from a systemd service has access only to systemds environment, which does not include things like SSH_AUTH_SOCK that you set in your .bash_profile or .zprofile.
In order to "import" this environment variable to systemd, you need to add systemctl --user set-environment SSH_AUTH_SOCK=$SSH_AUTH_SOCK to your .bash_profile as well.
However, only units loaded after this command will be able to see these environment variables. So you need to create a timer and a service, where the timer starts the service only after a period long enough to parse the .bash_profile, eg:

...
[Timer]
OnBootSec=10m
OnUnitActiveSec=1h
...

Then in the service, you don't need to do anything additional.

Alternatively, set a static socket address for your SSH-agent, as firecat53.

Thanks for everyones help.

Arch Linux

#1 2016-06-07 15:11:53

[SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#2 2016-06-07 15:18:06

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#3 2016-06-07 15:23:44

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#4 2016-06-07 15:31:43

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#5 2016-06-07 15:47:27

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#6 2016-06-07 16:16:15

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#7 2016-06-07 17:00:31

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#8 2016-06-07 20:54:23

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#9 2016-06-07 23:47:33

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

#10 2016-06-08 15:25:42

Re: [SOLVED]Invoking duplicity (sftp) via systemd fails but works manually

Board footer