You are not logged in.
Hello!
I'm trying to automate incremental back ups of $HOME on my personal computer. Rsnaphot seems to be the right tool for that, but I'm puzzled by the scheduling. Indeed, my computer is shut down and restarted every day, often several time a day, and this will mess up all the rotation process, I presume.
Indeed, the wiki page of rsnapshot suggests to schedule daily rotation of back ups by means of a systemd service /etc/systemd/system/rsnapshot-daily.timer:
[Unit]
Description=rsnapshot daily backup[Timer]
# 05:30 is the clock time when to start it
OnCalendar=05:30
Persistent=true
Unit=rsnapshot@daily.service[Install]
WantedBy=timers.target
But since my computer will be down at 05:30, this shouldn't work. In addition, for speed up reason, I would like to avoid any back up at boot time (in case I turn on my computer just at 05:30 for example).
Is there a way to run automatically 'rsnapshot daily' either:
- 10 minutes after each first boot of the day, or;
- when the previous daily back up is more than 24h old, or;
- (more cumbersone, I guess) when the computer has been used 24h times since the last rotation?
Best,
Last edited by LuX (2021-07-03 15:38:15)
Offline
man systemd.timer | less -p OnBootSecOffline
So, systemd.timer has a man page. Many thanks for pointing out!
This page gives instructions, but nevertheless a few more examples would be helpfull too.
I have created /etc/systemd/system/rsnapshot-hourly.timer with the following content. Can anyone confirm that this will:
- start the unit 10 minutes after booting (this is clear), and;
- restart it 40, 70, 100... and so on, minutes after booting (this is less clear to me)
--- EDIT : After some testing with shorter intervals of time, I can confirm that it works as expected. ---
[Unit]
Description=rsnapshot hourly backup[Timer]
# Start first 10 minutes after booting.
OnBootSec=10min
# Re-start then every 30 minutes later, and so on.
OnUnitActiveSec=30min
Unit=rsnapshot@hourly.service[Install]
WantedBy=timers.target
--- EDIT : New question below ---
Also, I have created /etc/systemd/system/rsnapshot-daily.timer below as in the wiki.
[Unit]
Description=rsnapshot daily backup[Timer]
# 05:30 is the clock time when to start it
OnCalendar=05:30
Persistent=true
Unit=rsnapshot@daily.service[Install]
WantedBy=timers.target
The man page says that "when the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive".
After some tests, it appears that it doesn't work as I understand it: I have replaced 05:30 in "OnCalendar=05:30" by some hour HH:MM, enabled both daily and hourly services, restarted my computer a few minutes before HH:MM, and waited a few minutes after HH:MM. Nothing happened. The hourly.[0-9] directories were created and filled as expected, but the daily.0 directory was not created at HH:MM.
Before HH:MM I had:
$ systemctl status rsnapshot-daily.timer
● rsnapshot-daily.timer - rsnapshot daily backup
Loaded: loaded (/etc/systemd/system/rsnapshot-daily.timer; enabled; vendor preset: disa>
Active: active (waiting) since Fri 2021-07-02 15:26:21 CEST; 56s ago
Trigger: Fri 2021-07-02 15:32:00 CEST; 4min 42s left
Triggers: ● rsnapshot@daily.servicejuil. 02 15:26:21 seth systemd[1]: Started rsnapshot daily backup.
After HH:MM I had:
$ systemctl status rsnapshot-daily.timer
● rsnapshot-daily.timer - rsnapshot daily backup
Loaded: loaded (/etc/systemd/system/rsnapshot-daily.timer; enabled; vendor preset: disa>
Active: active (waiting) since Fri 2021-07-02 15:26:21 CEST; 6min ago
Trigger: Sat 2021-07-03 15:32:00 CEST; 23h left
Triggers: ● rsnapshot@daily.servicejuil. 02 15:26:21 seth systemd[1]: Started rsnapshot daily backup.
What am I missing here?
Last edited by LuX (2021-07-02 14:07:59)
Offline
systemctl list-timerswill give you some insight when timers are going to run. It will show you the last and the next timer run times.
For daily snapshots, only once a day and not running immediately on boot, do something like:
[Timer]
OnCalendar=02:00
AccuracySec=30min
Persistent=trueThat is, without OnBootSec=.
AccuracySec= is there to delay the timer on boot.
Offline
"systemctl list-timers" says that rsnaphot-daily actualy ran, apparently. I don't understand why hourly.0 directory was not created, though. It is created when I run "rsnapshot daily" by hand.
Nevertheless, I will follow your advice and add "AccuracySec=MM" with various MM in rsnaphot-[daily|weekly|monthly].timer. I will use it in real conditions for some weeks (with the real schedules and directories) and see what happens.
Thank you for your help. ![]()
PS: I mark the discussion as SOLVED, but will unmark it if some problem arise in the next weeks.
Last edited by LuX (2021-07-02 16:10:16)
Offline
You still run the risk that a second rsnapshot instance will try to start a backup. It will fail, because rsnapshot uses a lock during a backup.
What you can do is to run all the rsnapshot intervals sequentually.
Something like this:
RSNAPSHOTDIR=...
# Reverse time order
echo "$(date +'%Y-%m-%d %H:%M'): INFO: Started rsnapshot" > "$RSNAPSHOTDIR/rsnapshot.log"
if [ "$(date +%-u)" -eq 6 ] && [ "$(date +%-d)" -le 07 ]; then
rsnapshot.sh -c /etc/rsnapshot.conf monthly | grep -v "not present (yet), nothing to "
elif [ "$(date +%-u)" -eq 6 ]; then
rsnapshot.sh -c /etc/rsnapshot.conf weekly | grep -v "not present (yet), nothing to "
else
rsnapshot.sh -c /etc/rsnapshot.conf daily
fi
echo "$(date +'%Y-%m-%d %H:%M'): INFO: Finished rsnapshot" >> "$RSNAPSHOTDIR/rsnapshot.log"in a script invoked by a daily running service.
Offline
I don't know if it is because of a conflict between two rsnapshot instances as suggested by willemw (honestly, I don't think so, not always), but it is a matter of fact that it doesn't work. The first hourly snapshot worked, and that's all. It is supposed to increment the hourly snapshots every 30mns, but it didn't, even after 3 hours.
$ systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sat 2021-07-03 00:00:00 CEST 2h 17min left Fri 2021-07-02 06:02:13 CEST 15h ago man-db.timer man-db.service
Sat 2021-07-03 00:00:00 CEST 2h 17min left Fri 2021-07-02 06:02:13 CEST 15h ago shadow.timer shadow.service
Sat 2021-07-03 05:30:00 CEST 7h left Fri 2021-07-02 16:59:00 CEST 4h 43min ago rsnapshot-daily.timer rsnapshot@daily.service
Sat 2021-07-03 18:28:30 CEST 20h left Fri 2021-07-02 18:28:30 CEST 3h 14min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
n/a n/a Fri 2021-07-02 18:33:30 CEST 3h 9min ago rsnapshot-hourly.timer rsnapshot@hourly.service
I have just uncommented the line in /etc/rsnapshot.conf which allows writing log files. I'm going to reboot and see if I can get some information from it.
@willemw: Thank you for the script. I will use it if I can't make any progress on this.
Offline
Are all the relevant "retain" lines uncommented in rsnapshot.conf?
If you, in the script, echo/printf to stdout/stderr instead of to a log file, then I think the output will be logged in journald. You can monitor the logs by, for instance:
journalctl -f -u rsnapshot-daily.serviceand test by waiting for the timer or by running:
sudo systemctl start rsnapshot-daily.serviceThere is also /var/log/rsnapshot.
Offline
@willemw: Yes, the retain lines are uncommented.
$ cat /etc/rsnapshot.conf | grep retain
retain hourly 9
retain daily 7
retain weekly 4
retain monthly 9
Nothing relevant was written into /var/log/rsnapshot. It ends after the first (and only) hourly rsnapshot without reporting any error:
[2021-07-02T22:25:49] /usr/bin/rsync -a --safe-links --one-file-system --delete --numeric-ids --relative --delete-excluded --exclude=/*.iso --exclude=/**/.cache/* --exclude=/**/.thumbnails/* --exclude=/**/Trash/* --link-dest=/mnt/tera1/snapshots/hourly.1/arch/ /etc/ /mnt/tera1/snapshots/hourly.0/arch/
[2021-07-02T22:25:50] touch /mnt/tera1/snapshots/hourly.0/
[2021-07-02T22:25:50] rm -f /var/run/rsnapshot.pid
[2021-07-02T22:25:50] /usr/bin/rsnapshot hourly: completed successfully
On the other hand I have the following warning:
$ systemctl status rsnapshot-hourly.timer
● rsnapshot-hourly.timer - rsnapshot hourly backup
Loaded: loaded (/etc/systemd/system/rsnapshot-hourly.timer; enabled; vendor preset: disabled)
Active: active (elapsed) since Fri 2021-07-02 22:05:23 CEST; 2h 6min ago
Trigger: n/a
Triggers: ● rsnapshot@hourly.servicejuil. 02 22:05:23 seth systemd[1]: Started rsnapshot hourly backup.
juil. 02 22:59:57 seth systemd[1]: /etc/systemd/system/rsnapshot-hourly.timer:1: Assignment outside of section. Ignoring.
juil. 02 22:59:57 seth systemd[1]: /etc/systemd/system/rsnapshot-hourly.timer:10: Failed to parse timer value, ignoring: 30mn
juil. 03 00:12:12 seth systemd[1]: /etc/systemd/system/rsnapshot-hourly.timer:1: Assignment outside of section. Ignoring.
Nevertheless, running "rsnapshot hourly" after that does the job smoothly (which proves that /etc/rsnapshot.conf is fine).
Any suggestion?
Offline
Could try increasing the "verbose" level in rsnapshot.conf.
Should be 30min. (Check: systemd-analyze timespan 30mn).
It looks like the lines in the .timer files are not all inside a section or correct section ([Unit], [Timer], ...).
Note: the "started"/"finished" echo statements in my if-then-else example are only additional logging for convenience. They can be left out. The output of rsnapshot will still be logged.
Offline
I have removed all the *.timer files and the previous snapshots, to restart from scratch without OnBoot=, OnUnitActiveSec= and AccuracySec= lines in the *.timer files.
The *[daily|weekly|monthly].timer files are the same as in the wiki. The *hourly.timer file is almost the same, but there are 3 OnCalendar= lines:
$ cat /etc/systemd/system/rsnapshot-hourly.timer
/etc/systemd/system/rsnapshot-hourly.timer[Unit]
Description=rsnapshot hourly backup[Timer]
# Se lance toutes les 20mn.
OnCalendar=*-*-* *:00:00
OnCalendar=*-*-* *:20:00
OnCalendar=*-*-* *:40:00
Persistent=true
Unit=rsnapshot@hourly.service[Install]
WantedBy=timers.target
It works like a charm: hourly.[0-9] directories are created/rotated every 20minutes (at HH:00, HH:20, HH:40). I don't know yet if the daily.[0-9] directories will be created too, but I'm optimistic.
It seems that these OnBoot=, OnUnitActiveSec= and AccuracySec= lines were confusing, although I can't see any reason why.
I still have this error, though:
$ systemctl status rsnapshot-hourly.timer
* rsnapshot-hourly.timer - rsnapshot hourly backup
Loaded: loaded (/etc/systemd/system/rsnapshot-hourly.timer; enabled; vendor preset: disabled)
Active: active (waiting) since Sat 2021-07-03 10:36:26 CEST; 1h 40min ago
Trigger: Sat 2021-07-03 12:20:00 CEST; 3min 16s left
Triggers: * rsnapshot@hourly.serviceJul 03 10:36:26 seth systemd[1]: Started rsnapshot hourly backup.
Jul 03 10:37:42 seth systemd[1]: /etc/systemd/system/rsnapshot-hourly.timer:1: Assignment outside of section. Ignoring.
Any idea?
It looks like the lines in the .timer files are not all inside a section or correct section ([Unit], [Timer], ...).
Could you tell me if my *hourly.timer file above is correct?
Could try increasing the "verbose" level in rsnapshot.conf.
Should be 30min. (Check: systemd-analyze timespan 30mn).
I have set the verbosity level to 5. The /var/log/rsnapshot file is now 96390 lines long, but doesn't report any warning or error. The systemd-analyse command doesn't work.
$ LANG=C ; systemd-analyze timespan 30mn
Failed to parse time span '30mn': Invalid argument
I am tempted to keep it as is, if the daily rotation works tomorrow. Anyway, if sombody can explain the errors I had previously with the OnBoot= and OnUnitActiveSec= lines, I would be interrested.
Note: The "Automation" section of the wiki which gives these working examples of *[hourly|daily|weekly|monthly].timer files is marked as "needing language" because "Too much code for an article". I disagree: such examples are pretty much useful, and adding more working examples for different situations would even be better. A reference to the man page of systemd.timer could be usefull too.
Offline
Wow, so many timers in that wiki and only for one program. I only have only 3 custom timers (and 3 scripts) I made myself on my systems: boot.timer, hourly.timer and daily.timer. The daily script also does weekly and monthly tasks. Of course I have more timers running.
/etc/systemd/system/rsnapshot-hourly.timer:1means line 1 is incorrect. Remove that line. That "/etc/systemd/system/rsnapshot-hourly.timer" is the header of that code section (in BBCode), it is not part of the file content. In HTML you cannot see the difference between the header and the content of a code section.
Change 30mn to 30min or 30m.
I think you can reduce all the OnCalendar= lines to: OnCalendar=*-*-* *:*/20:00
*/20 meaning every 20 minutes.
See https://www.freedesktop.org/software/sy … .time.html for more time examples.
Never used OnUnitActiveSec=. See my [Timer] example above. Used that for years without problems.
My hourly timer is something like this (actually runs once every 2 hours):
[Unit]
Description=Run hourly
After=boot.timer
[Timer]
# Every two hours
#OnCalendar=00/2:00:00
OnCalendar=hourly
AccuracySec=30min
Persistent=true
[Install]
WantedBy=timers.targetStarting from scratch, I would first only test the .service file (sudo systemctl start ...service) and make that working. Then enable 1 timer and make that working.
Offline
Wow, so many timers in that wiki and only for one program. I only have only 3 custom timers (and 3 scripts) I made myself on my systems: boot.timer, hourly.timer and daily.timer. The daily script also does weekly and monthly tasks.
Many timers for one program, that's true (although 4 is not really a large number...), but rsnapshot is a program which needs several timers by essence. So this section of the wiki seems consistent to me (and helpfull).
On the other hand, this section might be reduced if a reference to the wiki page dedicated to systemd timers would be added. This would be a good idea anyway: it is the first time I'm using timers, only because they are required by rsnapshot, so I didn't imagine that timers could be a subject in itself, with it's own wiki page and man page, until now. I guess that other newbies may think the same.
/etc/systemd/system/rsnapshot-hourly.timer:1means line 1 is incorrect. Remove that line. That "/etc/systemd/system/rsnapshot-hourly.timer" is the header of that code section (in BBCode), it is not part of the file content. In HTML you cannot see the difference between the header and the content of a code section.
You're right, captain obvious! ![]()
I did not even SEE this line above [Unit] because it is not part of the structure. Funny...
Thank you very much for your help. I'll mark this topic as SOLVED.
--- EDIT July, 4th ---
Note: Even if I'm not doing anything to avoid snapshots services to run at boot time, it seems that systemd always starts them at least a few minutes after the boot. So there's no need to OnBoot= and OnUnitActiveSec= lines for me (apparently).
Not at all! With the timers given in the wiki, when I shut down my computer, and restart it, if the time of a snapshot has passed in the mean time, systemd runs this snapshot during the boot, which suspends it for a long time. No good.
So using OnBoot= is mandatory. But this option has other side effects which I would prefer to avoid (see below).
Last edited by LuX (2021-07-04 17:16:41)
Offline
It's good to learn about timers and use them. But you can also run rsnapshot from cron, the "old" way.
That multiple timer approach in the rsnapshot Wiki: it introduces, unnecessarily in my opinion, possible backup conflicts (backups in parallel).
Other backup programs I know of that provide hourly, daily, weekly, etc. backups (Btrbk, Sanoid), they only provide a single timer (and a single service) and therefore no backup conflicts from .timers files are possible. For rsnapshot, that if-then-else example I gave above is a way to avoid backup conflicts.
Offline
Hum... you are meaning that systemd won't be aware that these different timers related to rsnapshot have a hierarchy, right?
I heard that systemd is precisely a smart way to deal with several processes, possibly contestant. It would be disappointing if it cannot deal properly with several timers on the same program. But even if it is so, /etc/rsnapsot.conf has the following option, which is enabled by default. Doesn't it prevent what you're talking about?
# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile /var/run/rsnapshot.pid
Last edited by LuX (2021-07-03 15:44:13)
Offline
systemd only knows about its units (.service files, etc.), not about the rsnapshot program. Only the exit return code of a program.
I already mentioned rsnapshot lock. If there is a conflict, the second instance of rsnapshot will fail. The changes are small, I know.
For example, if your monthly backup fails and weekly backups get pruned after that, then your oldest backup may be less than a month old. Then your backups don't meet your expectations/requirements.
Offline
@willemw: You convinced me!
In the mean time I have experienced that (contrary to my first impression), with the timers in the wiki it happens sometimes (quite often, actually) that a snapshot is run during the boot, which suspends it during more than a minute. So the OnBoot= option is mandatory. But then, if I shut down and restart my computer 4 times in 20 minutes for exemple (which can happen if I have a problem, or if I'm doing some test), a hourly snapshot will be done every time I reboot, which is not the expected behaviour.
So I'm considering to write a script, call it 5 minutes after the boot with a single timer, and let it manage the various calls to rsnapshot hourly|daily|weekly|monthly (while doing several tests, not only on the current time, but also on the timestamp of the last snapshots, which I couldn't do only with timers). This is what you suggested, but the example script you gave is only a starting point for what I'm aiming at.
By the way, what is
rsnapshot.sh -c /etc/rsnapshot.conf monthlyin your script? Shouldn't it be
/usr/bin/rsnapshot monthlyinstead?
Last edited by LuX (2021-07-04 17:13:58)
Offline
The automation example in the rsnapshot Wiki is fine, I guess, but not ideal. Maybe run conflicts can be avoided with only using .timer and .service files (After=, Before=, ...).
What is suspended for more than one minute?
Are you going to add yet another timer (5min after boot)? And check the rsnapshot timestamps? Sounds a bit overly complex.
How do you know a backup completed successfully and was not interrupted? Are you going to test that as well?
Create the backup manually the first time (systemctl start), so that subsequent backups will take less time.
I only do daily, weekly, monthly backups with a script that runs uninterrupted once a day during the night. No hourly backups or backups immediately after boot.
But I do have other hourly tasks (with a similar if-then-else statement as above). If a (single) task should not start immediately after boot, then I currently solve that by putting a "sleep 2m" before running that task. This is done inside a script. I am not adding any timers. You could maybe do the same.
There is one issue. I have "After=boot.timer" in my hourly.timer and my boot.timer is something like:
[Unit]
Description=Run at boot up
[Timer]
OnBootSec=4min
#AccuracySec=1min
Persistent=true
[Install]
WantedBy=timers.targetso in theory that "sleep 2m" should not be necessary because of the "After=boot.timer", I would have thought. The first hourly service run should not start before 4 minutes after boot. Let my know if you find a solution without adding another timer.
Note that only boot.timer has "OnBootSec=". The other timers should not have that, otherwise they will run at boot time (https://www.freedesktop.org/software/sy … timer.html).
"rsnapshot.sh -c /etc/rsnapshot.conf monthly" and "/usr/bin/rsnapshot monthly" are the same. Actually, I have "rsnapshot.sh -c /etc/rsnapshot-checksum.conf", which contains an extra "--checksum" on the "rsync_long_args" line.
Another thing. On the Arch rsnapshot Wiki: why is there "IOSchedulingClass=idle"?
IOPRIO_CLASS_IDLE (3)
This is the idle scheduling class. Processes running at
this level get I/O time only when no one else needs the
disk. The idle class has no class data. Attention is
required when assigning this priority class to a process,
since it may become starved if higher priority processes
are constantly accessing the disk.Is this not bad if you want do to something important like save your data to disk?
BTW, most of my backups are now on Btrfs and ZFS (snapshots and send/receive replication).
Last edited by willemw (2021-07-04 18:51:48)
Offline
Let my know if you find a solution without adding another timer.
I'll post my script here as soon as it is finished (and tested).
The point is to avoid all the previous mentioned problems:
- No rsnapshot during the boot.
- No conflict between rsnsapshot instances.
- No reduced time intervals between snapshots in case of frequent reboots.
The script is working, and does not require any timer at all. But I still need to improve it a little bit.
Another thing. On the Arch rsnapshot Wiki: why is there "IOSchedulingClass=idle"?
I have no idea on this.
But I do have another question. My script is run on boot as a service. It will later call rsnapshot repeatedly (and sequentially, so no more conflict), not as a service, but as a command: '/usr/bin/rsnapshot hourly|daily|etc'. What happens if I shutdown the computer while rsnapshot is running? I guess that it could mess up the snapshots. How can I teach systemd to delay the shut down until the current instance of rsnapshot is finished?
BTW, most of my backups are now on Btrfs and ZFS (snapshots and send/receive replication).
I did't know about Btrfs and ZFS. This Bonwick's blog article comparing the (non-)limitations of ZFS with the mass/energy of the ocean, is amazing. ![]()
Offline
For memory, running rsnapshot with just the timers of the wiki can cause some delay at boot but also some errors. When I turned on my PC this morning, it must have tried to run rsnapshot-daily service for the first time, which caused a long delay, after which the following error message (still readable on tty1) was written.
Starting version 248.3-2-arch
arch: clean, 306306/1281120 files, 2483191/512000 blocks
[ TIME ] Timed out waiting for device /dev/sdb4.
[DEPEND] Dependency failed for /dev/sdb4.
[DEPEND] Dependency failed for Swap.
[FAILED] Failed to start rsnapshot (daily) backup.The daily.0 directory is not yet created (although it should be), which confirms the above message.
This is to say that some work-around is needed (using OnBoot= or other options in timers), if not using a devoted script as you suggested.
Offline
I would set it to IOSchedulingClass=low or not set it at all.
No hourly backups avoid most of these issues. The only backups I do during the day are manual backups and by a pacman hook which runs before an install, etc.
Ext4 is fine.
Btrfs and ZFS, they checksum all the files. With other filesystem, your files may be corrupt and you won't know about it. Your backups will also contain the same corrupt files.
Also, with Btrfs and ZFS, you don't have to plan your partition sizes beforehand.
For backups with a large amount of (small) files, Btrfs/ZFS snapshots and send/receive replication are so much better. Must faster and far less harddisk head seeking (movements). Also less (network) I/O.
I think rsnapshot creates the new hourly.0, daily.0, etc. snapshot only after a successful backup. Not sure which folder is the one used during backup. But there are different scenarios (local filesystem, rsync), so maybe don't rely on this fact.
Have not used it because I only do nightly backups, but there is also "rsnapshot sync" and the "sync_first" option to stage the current backup: "This allows better recovery in the event that rsnapshot is interrupted in the middle of a sync operation, since the sync step and rotation steps are separated."
Or maybe making the lockfile persistent will help recognizing incomplete backups: lockfile /var/tmp/rsnapshot.pid
Offline
Have you added path conditions to your .service file? That's not mentioned in the rsnapshot Wiki. You could do that. Or do before calling rsnapshot:
DIR=...
if ! mountpoint --quiet "$DIR"; then
...
return
fiDon't use /dev/sd*. It can change after a reboot.
Offline
I'm answering to myself.
How can I teach systemd to delay the shut down until the current instance of rsnapshot is finished?
Following this discussion, systemd uses to wait for services to be stopped before shutting down. Hence it suffices to call rsnapshot as a service instead of a command (e.g. 'systemd start rsnapshot@hourly' instead of '/usr/bin/rsnapshot hourly). This is a good reason to do it, contrary to what I thought first.
Of course this won't help in case of hard reboot, but nothing can be done in that case, I'm afraid ('rsnapshot rsync' may reduce the damage, but probably not fully avoid it).
Last edited by LuX (2021-07-05 22:31:32)
Offline
No, you can only delay shutdown. Have you ever seen "stop jobs" during shutdown? Services will be killed after a timeout when shutting down.
/etc/systemd/system.conf
#DefaultTimeoutStopSec=90sOffline
However, I have seen with ZFS that a service can hang forever.
Offline