script started with systemd timer doesn't execute correctly

spendzo · 2014-10-28 22:24:54

Hi everybody,

so, i have a problem with script not executing correctly. Purpose of the script is to download bloclists and restart a service.
In script, first thing it is supposed to do is to send an email, which it does, then it is supposed to do what it does and the last thing is to send an email (completion notification).

When executed manually, it works fine. When systemd timer does it, well, some things don't work as they are expected to.

The most problematic thing i find is not sending email with completion notification.
Anyone know what the problem might be?

timer-daily.target

[Unit]
Description=Daily Timer Target
StopWhenUnneeded=yes

transmission-blocklist-update.service

[Unit]
Description=Starts transmission blocklist update script
Wants=timer-daily.timer

[Service]
Type=simple
ExecStart=/home/username/administration/scripts/transmission-blocklist-update.sh  


[Install]
WantedBy=timer-daily.target

timer-daily.timer

[Unit]
Description=Daily Timer

[Timer]
OnCalendar=daily
Persistent=true
Unit=timer-daily.target

[Install]
WantedBy=basic.target

/home/username/administration/scripts/transmission-blocklist-update.sh

#!/bin/bash
BLOCKLIST_DIR=/var/lib/transmission/.config/transmission-daemon/blocklists

mail -s "Starting transmission blocklist update..." me@mymail.com < /dev/null 

#wget -q -O- http://www.bluetack.co.uk/config/level1.gz | gunzip -c > ${BLOCKLIST_DIR}/level1
#wget -q -O- http://www.bluetack.co.uk/config/level2.gz | gunzip -c > ${BLOCKLIST_DIR}/level2
#wget -q -O- http://www.bluetack.co.uk/config/level3.gz | gunzip -c > ${BLOCKLIST_DIR}/level3

wget -q -O- http://list.iblocklist.com/lists/atma/atma | gunzip -c > ${BLOCKLIST_DIR}/atma
wget -q -O- http://list.iblocklist.com/lists/bluetack/ads-trackers-and-bad-pr0n | gunzip -c > ${BLOCKLIST_DIR}/ads-trackers-and-bad-pr0n
wget -q -O- http://list.iblocklist.com/lists/bluetack/bad-peers | gunzip -c > ${BLOCKLIST_DIR}/bad-peers
wget -q -O- http://list.iblocklist.com/lists/bluetack/dshield | gunzip -c > ${BLOCKLIST_DIR}/dshield
wget -q -O- http://list.iblocklist.com/lists/bluetack/level-1 | gunzip -c > ${BLOCKLIST_DIR}/lev1
wget -q -O- http://list.iblocklist.com/lists/bluetack/level-2 | gunzip -c > ${BLOCKLIST_DIR}/lev2
wget -q -O- http://list.iblocklist.com/lists/bluetack/level-3 | gunzip -c > ${BLOCKLIST_DIR}/lev3
wget -q -O- http://list.iblocklist.com/lists/bluetack/microsoft | gunzip -c > ${BLOCKLIST_DIR}/microsoft
wget -q -O- http://list.iblocklist.com/lists/bluetack/proxy | gunzip -c > ${BLOCKLIST_DIR}/proxy
wget -q -O- http://list.iblocklist.com/lists/bluetack/spider | gunzip -c > ${BLOCKLIST_DIR}/spider
wget -q -O- http://list.iblocklist.com/lists/bluetack/spyware | gunzip -c > ${BLOCKLIST_DIR}/spyware
wget -q -O- http://list.iblocklist.com/lists/bluetack/web-exploit | gunzip -c > ${BLOCKLIST_DIR}/web-exploit
wget -q -O- http://list.iblocklist.com/lists/dchubad/faker | gunzip -c > ${BLOCKLIST_DIR}/faker
wget -q -O- http://list.iblocklist.com/lists/dchubad/hacker | gunzip -c > ${BLOCKLIST_DIR}/hacker
wget -q -O- http://list.iblocklist.com/lists/dchubad/pedophiles | gunzip -c > ${BLOCKLIST_DIR}/pedophiles
wget -q -O- http://list.iblocklist.com/lists/dchubad/spammer | gunzip -c > ${BLOCKLIST_DIR}/spammer
wget -q -O- http://list.iblocklist.com/lists/tbg/bogon | gunzip -c > ${BLOCKLIST_DIR}/bogon
wget -q -O- http://list.iblocklist.com/lists/tbg/business-isps | gunzip -c > ${BLOCKLIST_DIR}/business-isps
wget -q -O- http://list.iblocklist.com/lists/tbg/educational-institutions | gunzip -c > ${BLOCKLIST_DIR}/educational-institutions
wget -q -O- http://list.iblocklist.com/lists/tbg/general-corporate-ranges | gunzip -c > ${BLOCKLIST_DIR}/general-corporate-ranges
wget -q -O- http://list.iblocklist.com/lists/tbg/hijacked | gunzip -c > ${BLOCKLIST_DIR}/hijacked
wget -q -O- http://list.iblocklist.com/lists/tbg/primary-threats | gunzip -c > ${BLOCKLIST_DIR}/primary-threats
wget -q -O- http://list.iblocklist.com/lists/tbg/search-engines | gunzip -c > ${BLOCKLIST_DIR}/search-engines

#paid

systemctl restart transmission

mail -s "Transmission blocklist update finished." me@mymail.com < /dev/null 

now=$(date +"%m_%d_%Y")
echo "Transmission update finished on $now" >> /home/username/administration/logs/transmission-update.log

branch · 2014-10-29 04:41:51

I have absolutely no idea. That said, here is what I would do to troubleshoot:

1) Stop the timer and start transmission-blocklist-update.service directly with "systemctl start transmission-blocklist-update.service". See if the behaviour is the same as when started with the timer. If so, all future troubleshooting should be by starting the service directly instead of via the timer.

2) Probably irrelevent, but change transmission-blocklist-update.service "Type" to "oneshot" because it better fits the use case.

3) Comment all lines in the script between the two mail lines. Assuming you get both mails there are two possibilities:
a. something timeout related, probably due to one of the wget's taking too long (unlikely, as Type=simple should not be subject to timeout)
b. calling "systemctl restart" from within the context of a service file is problematic (i suspect this is the case). Try commenting only this line to test.

4) If 3) b. is the problem, try changing to "systemctl reload", or sending SIGHUP directly from your script to the trasmission-daemon process.

TheSaint · 2014-10-29 05:03:55

Did you look at journalctl report ?

# journalctl -b |grep -iE  '.*transmission-blocklist-update.*'

spendzo · 2014-10-29 20:08:26

branch wrote:

I have absolutely no idea. That said, here is what I would do to troubleshoot:
1) Stop the timer and start transmission-blocklist-update.service directly with "systemctl start transmission-blocklist-update.service". See if the behaviour is the same as when started with the timer. If so, all future troubleshooting should be by starting the service directly instead of via the timer.
2) Probably irrelevent, but change transmission-blocklist-update.service "Type" to "oneshot" because it better fits the use case.
3) Comment all lines in the script between the two mail lines. Assuming you get both mails there are two possibilities:
a. something timeout related, probably due to one of the wget's taking too long (unlikely, as Type=simple should not be subject to timeout)
b. calling "systemctl restart" from within the context of a service file is problematic (i suspect this is the case). Try commenting only this line to test.
4) If 3) b. is the problem, try changing to "systemctl reload", or sending SIGHUP directly from your script to the trasmission-daemon process.

This is a good hint, thanks... will look into this
But.. according to systemd documentation, "systemctl reload" would make the service reload configuration (settings.json). And SIGHUP, would it not only stop a service? So to start transmission it I'd need to call "systemctl start transmission-daemon", which would bring me to the context problem again...

TheSaint wrote:

Did you look at journalctl report ?
# journalctl -b |grep -iE  '.*transmission-blocklist-update.*'

Yes, nothing useful. There are a bunch of wrongly defined IP ranges in those blocklists, so this is what gets printed. Maybe too much errors stops the script later on?

I forgot to mention, sometimes it sends the confirmation email, but recently it fails all the time.

TheSaint · 2014-10-30 05:50:27

spendzo wrote:

Yes, nothing useful. There are a bunch of wrongly defined IP ranges in those blocklists, so this is what gets printed. Maybe too much errors stops the script later on?

So you should do more logging in your script, unless the timer isn't even able to start your script. (permission problem, maybe)

Secondly, you might disclose some detail from your journal, for help purposes.

falconindy · 2014-10-30 12:18:34

Several things I notice:

1) Your unit must be "Type=oneshot". Type=simple is intended for long-running processes, which your script isn't.
2) Your script should pass --no-block to systemctl. Otherwise, you may cause a deadlock in systemd.
3) WantedBy=basic.target doesn't mean what you probably think it means. Just use multi-user.target in timer-daily.target (this unit isn't even started until after basic.target as is).
4) The Wants=daily.timer in your service is redundant.

serdotlinecho · 2014-10-30 13:12:36

I don't know whether this will work or not, disable and stop your timer/service and try this:

/etc/systemd/system/transmission-blocklist-update.service

[Unit]
Description=Starts transmission blocklist update script
Wants=network.target

[Service]
Type=oneshot
ExecStart=%h/administration/scripts/transmission-blocklist-update.sh  

[Install]
WantedBy=multi-user.target

/etc/systemd/system/transmission-blocklist-update.timer

[Unit]
Description=Timer for transmission-blocklist-update.service

[Timer]
OnCalendar=daily
AccuracySec=1min
Persistent=true

[Install]
WantedBy=multi-user.target

Enable/start the timer:

# systemctl enable transmission-blocklist-update.timer
# systemctl start transmission-blocklist-update.timer
# systemctl status transmission-blocklist-update.timer

Last edited by serdotlinecho (2014-10-30 13:13:40)

spendzo · 2014-10-30 20:28:13

Thanks everybody, it works! It's fixed by changing service type to oneshot and adding --no-block to systemctl call... but...
Now, when the service is started manually, i have to wait for the script to finish since the terminal is waiting. Is this going to negatively impact system stability (i.e. deadlock) when the timer starts it?

falconindy wrote:

1) Your unit must be "Type=oneshot". Type=simple is intended for long-running processes, which your script isn't.

This script can execute for up to half an hour, because downloads above are just a subset of the all downloads in the actual script. What is considered to be a long running process (hours, days)?

The manual says:

Behavior of oneshot is similar to simple; however, it is expected that the process has to exit before systemd starts follow-up units.

So it's meant for sequantialy starting services, like the 'old' ways?

TheSaint wrote:

So you should do more logging in your script, unless the timer isn't even able to start your script. (permission problem, maybe)
Secondly, you might disclose some detail from your journal, for help purposes.

Not sure if it's relevant now, but the journal doesn't contain any output from the script. Only transmission-daemon writes to the log when it starts and loads downloaded lists.

@serdotlinecho, thanks. I think i did try this a while back, but this seems to be one timer for one service and I'd like to have my timer reused.

Last edited by spendzo (2014-10-30 20:41:48)

falconindy · 2014-10-30 22:05:32

falconindy wrote:

This script can execute for up to half an hour, because downloads above are just a subset of the all downloads in the actual script. What is considered to be a long running process (hours, days)?

"Long running" means that the service starts up, serves requests, and eventually shuts down on receipt of SIGTERM (or other clean signal). It has nothing to do with the actual duration. Your service starts up, does work, shuts down. That isn't (by this definition) long running, regardless of how long it takes according to the wall clock.

Leonid.I · 2014-10-30 23:24:51

spendzo wrote:

Thanks everybody, it works! It's fixed by changing service type to oneshot and adding --no-block to systemctl call... but...
Now, when the service is started manually, i have to wait for the script to finish since the terminal is waiting. Is this going to negatively impact system stability (i.e. deadlock) when the timer starts it?

No, this is just systemd's way of handling on-calendar timers. I suggest you either make an on-boot timer (say 15 sec into boot) to trigger that 1st timer (not sure if this is possible, btw), or better, use cron. This way, systemd starts cron w/o delay and then cron takes care about your script. I am not sure what advantage systemd.timer brings compared to cron in your case. Also, why do you even need a call to systemctl from the script. Systemctl will just call transmission with an appropriate restart flag, no?

Last edited by Leonid.I (2014-10-30 23:26:31)

spendzo · 2014-10-31 09:17:58

Leonid.I wrote:

spendzo wrote:
Thanks everybody, it works! It's fixed by changing service type to oneshot and adding --no-block to systemctl call... but...
Now, when the service is started manually, i have to wait for the script to finish since the terminal is waiting. Is this going to negatively impact system stability (i.e. deadlock) when the timer starts it?
No, this is just systemd's way of handling on-calendar timers. I suggest you either make an on-boot timer (say 15 sec into boot) to trigger that 1st timer (not sure if this is possible, btw), or better, use cron. This way, systemd starts cron w/o delay and then cron takes care about your script. I am not sure what advantage systemd.timer brings compared to cron in your case. Also, why do you even need a call to systemctl from the script. Systemctl will just call transmission with an appropriate restart flag, no?

I'm insisting on this solution mostly for educational purposes - trying to do a simple thing the hard way (I had it working previously in cron, so it is a backup solution if i can't make systemd timer work).
From the end user perspective, there are a few benefits that i can recognize: functionality is included in system package and cron doesn't need to be installed (i.e. system wouldn't need an additional package) and better logging (even though it's not enabled in my case).

But you gave me an idea which i didn't consider before: maybe there is a way to shut down a service as a prerequisite and start a service after certain other service has been stopped.

Anyway, this all will have to wait - there was a power surge last night and it fried something in my pc which i can't yet identify.
The drives seem to be ok, which is a good thing

Last edited by spendzo (2014-10-31 09:22:15)

Arch Linux

#1 2014-10-28 22:24:54

script started with systemd timer doesn't execute correctly

#2 2014-10-29 04:41:51

Re: script started with systemd timer doesn't execute correctly

#3 2014-10-29 05:03:55

Re: script started with systemd timer doesn't execute correctly

#4 2014-10-29 20:08:26

Re: script started with systemd timer doesn't execute correctly

#5 2014-10-30 05:50:27

Re: script started with systemd timer doesn't execute correctly

#6 2014-10-30 12:18:34

Re: script started with systemd timer doesn't execute correctly

#7 2014-10-30 13:12:36

Re: script started with systemd timer doesn't execute correctly

#8 2014-10-30 20:28:13

Re: script started with systemd timer doesn't execute correctly

#9 2014-10-30 22:05:32

Re: script started with systemd timer doesn't execute correctly

#10 2014-10-30 23:24:51

Re: script started with systemd timer doesn't execute correctly

#11 2014-10-31 09:17:58

Re: script started with systemd timer doesn't execute correctly

Board footer