Arch Linux

vguarnaccia

Member

Registered: 2018-02-25

Posts: 2

Systemd service fails to connect to network

I am trying to setup a daily backup to rsync.net but on startup, the service fails to connect to the internet. The service uses borgmatic but the issue seems to be that ssh fails. I am running Arch on an Asus laptop, and I want the service to run automatically when I open my laptop the first time each day.

When I run

# systemctl start borgmatic.service

after logging in, the service runs successfully but when triggered by the timer, it fails. I tried following the advice on the Arch wiki and the freedesktop page with no luck.

borgmatic.service:

[Unit]
Description=borgmatic backup

After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/borgmatic

borgmatic.timer:

[Unit]
Description=Daily borgmatic backup

[Timer]
OnCalendar=daily
Persistent=true

# This didn't help
# AccuracySec=12h
# RandomizedDelaySec=5m

[Install]
WantedBy=timers.target

$ systemctl is-enabled NetworkManager-wait-online.service systemd-networkd-wait-online.service
enabled
disabled

Yesterday's log:

Jun 30 09:53:21 archer systemd[1]: Starting borgmatic backup...
Jun 30 09:53:22 archer borgmatic[26613]: Sat Jun 30 09:53:22 PDT 2018 - Starting a backup job.
Jun 30 09:53:23 archer borgmatic[26613]: Remote: ssh: Could not resolve hostname usw-s003.rsync.net: Temporary failure in name resolution
Jun 30 09:53:23 archer borgmatic[26613]: Connection closed by remote host. Is borg working on the server?
Jun 30 09:53:23 archer borgmatic[26613]: Sat Jun 30 09:53:23 PDT 2018 - Error while creating a backup.
Jun 30 09:53:23 archer borgmatic[26613]: Command '('borg', 'prune', 'rsync_dot_net:borg_repository', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '6', '--keep-yearly', '1', '--prefix', '{hostnam>
Jun 30 09:53:23 archer systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 09:53:23 archer systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jun 30 09:53:23 archer systemd[1]: Failed to start borgmatic backup.

What happened when I added my rsync.net server's IP address in hosts.

Jun 23 12:46:50 archer systemd[1]: Starting borgmatic backup...
Jun 23 12:46:51 archer borgmatic[22743]: Sat Jun 23 12:46:51 PDT 2018 - Starting a backup job.
Jun 23 12:46:52 archer borgmatic[22743]: Remote: ssh: connect to host usw-s003.rsync.net port 22: Network is unreachable
Jun 23 12:46:52 archer borgmatic[22743]: Connection closed by remote host. Is borg working on the server?
Jun 23 12:46:52 archer borgmatic[22743]: Sat Jun 23 12:46:52 PDT 2018 - Error while creating a backup.
Jun 23 12:46:52 archer borgmatic[22743]: Command '('borg', 'prune', 'rsync_dot_net:borg_repository', '--keep-daily', '7', '--keep-weekly', '4', '--keep-monthly', '6', '--keep-yearly', '1', '--prefix', '{hostnam>
Jun 23 12:46:52 archer systemd[1]: borgmatic.service: Main process exited, code=exited, status=1/FAILURE
Jun 23 12:46:52 archer systemd[1]: borgmatic.service: Failed with result 'exit-code'.
Jun 23 12:46:52 archer systemd[1]: Failed to start borgmatic backup.

I am not very similar with systemd but I tried my best to make sense of its manpages. I have not found a way to replicate the startup environment to test my .timer short of restarting my laptop each time.

Any help with debugging this issue or even how to mimic the early startup system would be greatly appreciated.

vguarnaccia

Member

Registered: 2018-02-25

Posts: 2

Re: Systemd service fails to connect to network

I have found a working solution. Because my system uses NetworkManager, the service which indicates whether my network is up is NetworkManager-wait-online.service. The command it runs is

$ nm-online -s -q --timeout=30

. Here is what the manpage says about -s:

Wait for NetworkManager startup to complete, rather than waiting for network specifically.

For my case, and I would imagine most laptops, I do not want that flag. I overrode it using

# systemctl edit NetworkManager-wait-online.service

And remembering to clear ExecStart=, wrote in

[Service]
ExecStart=
ExecStart=/usr/bin/nm-online -q --timeout=3600

3600 seconds, or 1 hour, is the maximum timeout allowed.

To validate my results, I ran two different commands. Either should work. Here is what I did with expected outputs:

$ systemctl cat NetworkManager-wait-online.service
# /usr/lib/systemd/system/NetworkManager-wait-online.service
[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requires=NetworkManager.service
After=NetworkManager.service
Before=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes

[Install]
WantedBy=network-online.target

# /etc/systemd/system/NetworkManager-wait-online.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/nm-online -q --timeout=3600

Or

$ systemctl show --property=ExecStart NetworkManager-wait-online.service
ExecStart={ path=/usr/bin/nm-online ; argv[]=/usr/bin/nm-online -q --timeout=3600 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }

Last edited by vguarnaccia (2018-08-19 22:40:22)