Systemd ExecStop= using a pidof something else [SOLVED]

graysky · 2020-10-18 22:03:12

Is it possible to direct a service to run "/bin/kill -TERM xxx" where xxx is the target PID? Note that the target PID is different from $MAINPID. I tried using "ExecStop=/bin/kill -TERM $(/usr/bin/pidof kodi-x11)" but that isn't parsed properly by systemd. I have been reading through the systemd man pages but am unable to find what I want. Ideas?

Journalctl output using my hacky pidof line:

kill[4583]: kill: cannot find process "kodi-x11)"

Complete service file as-is for reference:

[Unit]
Description=Kodi standalone (X11)
After=remote-fs.target systemd-user-sessions.service network-online.target nss-lookup.target sound.target bluetooth.target polkit.service upower.service mysqld.service
Wants=network-online.target polkit.service upower.service
Conflicts=getty@tty1.service

[Service]
User=kodi
Group=kodi
EnvironmentFile=-/etc/conf.d/kodi-standalone
PAMName=login
TTYPath=/dev/tty1
Environment=WINDOWING=x11
ExecStart=/usr/bin/xinit /usr/bin/kodi-standalone -- :0 -quiet -nolisten tcp vt1
Restart=on-abort
StandardInput=tty
StandardOutput=journal

[Install]
Alias=display-manager.service

Last edited by graysky (2020-10-24 17:21:54)

Awebb · 2020-10-18 23:32:40

Have you tried backticks instead of $()?

Edit: apparently, systemd doesn't fully implement bash. Have you tried a subshell, like with bash -c?

Edit2: also, deeplink: https://freedesktop.org/software/system … nd%20lines

Last edited by Awebb (2020-10-18 23:40:12)

Trilby · 2020-10-19 00:35:03

Didn't we learn in the last thread that killing kodi-x11 did not acheive what you wanted, but killing kodi-standalone was what was required?

graysky · 2020-10-19 10:27:38

@Awebb - Not sure what deeplink is in the man page you linked. I haven't tried backticks but based on your edit, am thinking they wouldn't work.

@Trilby - Yes, but this is to address another issue. Kodi stores some info to a config file on exit and data can be lost if we allow systemd to kill it or pkill to kill it. The right way is to exit kodi is with a SIGTERM and a brief pause. I am faking it currently using the cli python script (kodi-send) to send the "quit" signal which works fine.

Last edited by graysky (2020-10-19 10:32:17)

schard · 2020-10-19 10:39:39

You have an XY problem. Just use KillSignal= to your specs. https://www.freedesktop.org/software/sy … illSignal=

Update: I just realized that you don't actually run kodi in the systemd unit, but X.
I'd suggest that you change your architecture of systemd units accordingly, so that one unit runs the X-Server and another unit runs kodi on it.
This way you have full control over the respective programs without running into side effects like this.

Last edited by schard (2020-10-19 10:52:26)

graysky · 2020-10-19 12:12:11

schard wrote:

You have an XY problem. Just use KillSignal= to your specs. https://www.freedesktop.org/software/sy … illSignal=
Update: I just realized that you don't actually run kodi in the systemd unit, but X.
I'd suggest that you change your architecture of systemd units accordingly, so that one unit runs the X-Server and another unit runs kodi on it.
This way you have full control over the respective programs without running into side effects like this.

That is an interesting suggestion. I am wondering if getting the pidof kodi-x11 in this case however is a more simplistic solution.

progandy · 2020-10-19 13:25:44

Does kodi-x11 exit cleanly if it loses its connection to the X server, i.e. if only the x-server gets a SIGTERM signal? Then just set KillMode to mixed and it should work(TM).

If you want to execute more complex commands with ExecStop, you'll need to run them with a shell (bash, dash, ...)

Edit: Anyways, you don't need to call pidof, just use pkill with the -x / --exact parameter and --oldest or --newest instead. Som other options like limiting the kill command to a pgroup, session, or parent might be useful as well to ensure pkill doesn't reach a random process that coincidentally has the same name.

Last edited by progandy (2020-10-19 13:48:57)

graysky · 2020-10-19 15:24:46

@progandy - I am not entirely sure. Kodi should write out its cumulative uptime to ~/.kodi/userdata/guisettings.xml on a successful exit. With the service as-is, this is not happening. I do not know if it is because systemd kills it faster than the write out, or because killing xinit ends in a different state for kodi, or both.

I am happy to test out some configurations for pkill which I will do when I have some time to explore. If you have a favorite suggestion, please let me know (perhaps --newest)?

EDIT: Sending `/usr/bin/pkill --newest kodi-x11` from the CLI while the service is running gives the intended effect. I will code it into the formal service when I have some time and test it. Thanks for the suggestion :thumbs up:

Last edited by graysky (2020-10-19 15:55:26)

progandy · 2020-10-19 16:47:19

graysky wrote:

@progandy - I am not entirely sure. Kodi should write out its cumulative uptime to ~/.kodi/userdata/guisettings.xml on a successful exit. With the service as-is, this is not happening. I do not know if it is because systemd kills it faster than the write out, or because killing xinit ends in a different state for kodi, or both.

Systemd has different kill modes (see manpage systemd.kill) The default is "control group" which will send SIGTERM to all subprocesses of the service. Then after a timeout (default 90s I think) it will send SIGKILL to all of them. If you choose "mixed", then only $MAINPID will get SIGTERM, systemd expects that to stop all other processes. After the timeout all processes still get SIGKILL if they haven't stopped.
I suggested changing to the "mixed" mode and testing it.

Trilby · 2020-10-19 17:04:46

We effectively tested this in a previous thread. A sighup sent to the kodi processes was reportedly not sufficient for a clean exit. If systemd sends a sigterm to xinit only, then xinit will respond by sending a sighup to the kodi process.

progandy · 2020-10-19 17:17:03

Ah, thanks.

graysky · 2020-10-19 18:19:57

Thanks for the engagement, all. My initial testing shows that an ExecStop calling pkill --latest kodi-x11 per progandy's suggestion AND adding a sleep 1s to the command is needed to allow for the desired behavior. For a complete service, see: https://github.com/graysky2/kodi-standa … 11.service

I think this one is solved unless folks think my implementation is too hacky and have a more technically correct suggestion.

Trilby · 2020-10-19 18:53:08

The 1s is really ugly - what happens when it takes more than 1s to shutdown? Also, if this works, `xi` really should have worked as it sends a sigterm to the child process and then does a waitpid for the child to finish up (which is the clean and sane way of doing what you are sometimes getting with the sleep).

graysky · 2020-10-19 19:56:09

@Trilby - Yes, one key was noticing that the original ExecStart= line called a sh wrapper, kodi-standalone which in turn launched kodi-x11. The 1 sec should be all it needs. My understanding from a kodi developer is there is a slow 500 ms loop that listens for the SIGTERM. Since the write out to that file is fast, it shouldn't take longer. I suppose it could if the system is under load. I don't have a better option.

There are 2 goals:

1) Insure that the data on shutdown get written out.
2) Exit with a return code that reflects reality.

Note that kodi-x11 with xinit is just 1 of 3 use cases/services. There are also separate services for kodi-gbm (more simplistic without the ExecStop= at all) and kodi-wayland. See them here: https://github.com/graysky2/kodi-standa … 1b422/init

Trilby · 2020-10-19 23:13:51

graysky wrote:

My understanding from a kodi developer is there is a slow 500 ms loop that listens for the SIGTERM.

Signals are interrupts. The handler is called instantly regardless of the polling rate of any other processing of the code, and any sane loop would retest the loop condition after continuing from a signal (this is the case for all code I've ever seen that runs an event loop - e.g., using select, etc). So the 500ms shouldn't really be relevant - only the time actually needed to write to the file should be relevant. But I guess if the sleep works, go with it.

graysky · 2020-10-24 14:48:01

It could be that kodi takes longer than 1 sec to shut down.

A single ExecStop a hard-coded sleep high number can cover this but that could lead to a race condition if kodi is still busy:

ExecStop=/usr/bin/pkill --newest kodi-x11 ; /usr/bin/sleep 15s

What is the the correct way to write the service so that systemd will wait for a specific PID to exit on its own (defined by pgrep --newest kodi-x11)?

Last edited by graysky (2020-10-24 14:50:04)

Trilby · 2020-10-24 15:14:04

I don't know of any simple shell utilities that implement waitpid - but in C code that's what it would be.

graysky · 2020-10-24 15:25:18

I clearly do not understand systemd. The timeout binary provided by coreutils does this.

If I start the kodi-x11.service and then as root:

# timeout 8 /usr/bin/pkill --newest kodi-x11

I get the desired result as seen by running `tail -f /var/lib/kodi/.kodi/temp/kodi.log` in another shell. There, you see kodi do all its housekeeping tasks upon receiving the SIGTERM and then that PID disappears:

...
2020-10-24 10:00:56.243 T:139740558294848  NOTICE: Quitting due to POSIX signal
2020-10-24 10:00:56.283 T:139740558294848  NOTICE: Stopping player
2020-10-24 10:00:56.283 T:139740558294848  NOTICE: Storing total System Uptime
2020-10-24 10:00:56.283 T:139740558294848  NOTICE: Saving settings
2020-10-24 10:00:56.285 T:139740558294848  NOTICE: Saving skin settings
2020-10-24 10:00:56.285 T:139740558294848  NOTICE: Stopping all
2020-10-24 10:00:56.285 T:139740558294848  NOTICE: ES: Stopping event server
2020-10-24 10:00:56.285 T:139740558294848  NOTICE: CWebServer[8080]: Stopped
2020-10-24 10:00:57.280 T:139739092997696  NOTICE: ES: UDP Event server stopped
2020-10-24 10:00:57.889 T:139740558294848  NOTICE: Application stopped
2020-10-24 10:00:58.089 T:139740558294848  NOTICE: XBApplicationEx: destroying...
2020-10-24 10:00:58.100 T:139740558294848  NOTICE: unload skin
2020-10-24 10:00:58.116 T:139740558294848  NOTICE: unload sections
2020-10-24 10:00:58.170 T:139740558294848  NOTICE: XBApplicationEx: application stopped!

By contrast, when I use the same command as the ExecStop line, systemd does not seem to wait for the PID to go away on its own at all. If I tail the same log file, none of the house keeping tasks are allowed to run. What am I missing?

ExecStop=/usr/bin/timeout 8 /usr/bin/pkill --newest kodi-x11

Last edited by graysky (2020-10-24 15:25:37)

Trilby · 2020-10-24 15:31:29

I'm not sure how that test illustrates anything. First the timeout is on the pkill process: pkill returns immediately in both cases. Pkill does not wait for kodi-x11 to finish it's cleanup (and neither does the "timeout"). The only way "timeout" would change the behavior was if pkill itself took more than 8 seconds to finish which is just never the case (even if kodi-x11 would take more than 8 seconds to finish cleaning up and exit*). So using timeout here is a no-op. All you've done is the same test you've done before: when you run pkill on kodi-x11 *before* the service shuts down it gets ample time to do all of it's cleanup; but when pkill is run by the systemd service, there is not enough time. So again, I don't see how "timeout" adds or changes anything here.

*note: this is almost certainly the real problem: pkill sends the signal to kodi-x11 and then pkill itself exits immediately effectively saying "I'm done, I sent my signal" - but pkill does not wait for kodi-x11 to finish handling that signal. If there were a version of pkill that sent the signal then called a waitpid() on the pid it sent it too, the problem would most likely be solved.

Last edited by Trilby (2020-10-24 15:34:16)

progandy · 2020-10-24 16:03:25

killall has a --wait parameter, that could be used as well (I'd add the --user parameter as well). If kodi hangs and doesn't quit, then killall will wait endlessly until it is killed by the systemd-internal timeout.

ExecStop=/usr/bin/killall --user kodi --exact --wait kodi-x11

Last edited by progandy (2020-10-24 16:05:01)

Trilby · 2020-10-24 16:25:37

Ah cool, I didn't know about that flag for killall - though it's implementation seems odd:

man killall wrote:

killall checks once per second if any of the killed processes still exist ...

That'll get the job done, but I find it odd that they'd implement this through periodic checks rather than a wait()/waitpid().

graysky · 2020-10-24 17:11:29

@Trilby - Thank you for the explanation. Makes sense (discussion about pkill/timeout).
@progandy - Thank you for that suggestion. It appears to give the desired effect.

Last edited by graysky (2020-10-24 17:18:36)

Arch Linux

#1 2020-10-18 22:03:12

Systemd ExecStop= using a pidof something else [SOLVED]

#2 2020-10-18 23:32:40

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#3 2020-10-19 00:35:03

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#4 2020-10-19 10:27:38

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#5 2020-10-19 10:39:39

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#6 2020-10-19 12:12:11

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#7 2020-10-19 13:25:44

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#8 2020-10-19 15:24:46

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#9 2020-10-19 16:47:19

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#10 2020-10-19 17:04:46

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#11 2020-10-19 17:17:03

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#12 2020-10-19 18:19:57

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#13 2020-10-19 18:53:08

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#14 2020-10-19 19:56:09

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#15 2020-10-19 23:13:51

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#16 2020-10-24 14:48:01

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#17 2020-10-24 15:14:04

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#18 2020-10-24 15:25:18

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#19 2020-10-24 15:31:29

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#20 2020-10-24 16:03:25

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#21 2020-10-24 16:25:37

Re: Systemd ExecStop= using a pidof something else [SOLVED]

#22 2020-10-24 17:11:29

Re: Systemd ExecStop= using a pidof something else [SOLVED]

Board footer