Tweak a systemd unit for kodi-standalone to allow for a clean exit

graysky · 2020-09-21 15:05:49

Per a conversation with Trilby in email, this thread is to brainstorm some ideas to get a clean exit code when systemctl is called to stop kodi.service which I am providing here. Everything currently works with the code as is with the exception of a cosmetic unclean exit when the user calls systemctl to stop the service. Example:

● kodi.service - Kodi standalone (X11)
     Loaded: loaded (/usr/lib/systemd/system/kodi.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2020-09-21 10:04:19 CDT; 20s ago
    Process: 119176 ExecStart=/usr/bin/xinit /usr/bin/kodi-standalone -- :0 -nolisten tcp vt1 (code=exited, status=1/FAILURE)
   Main PID: 119176 (code=exited, status=1/FAILURE)

Sep 21 08:19:41 4legs systemd[1]: Started Kodi standalone (X11).
Sep 21 08:19:41 4legs systemd[119176]: pam_systemd_home(login:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Sep 21 08:19:41 4legs systemd[119176]: pam_unix(login:session): session opened for user kodi(uid=989) by (uid=0)
Sep 21 10:04:18 4legs systemd[1]: Stopping Kodi standalone (X11)...
Sep 21 10:04:19 4legs systemd[1]: kodi.service: Main process exited, code=exited, status=1/FAILURE
Sep 21 10:04:19 4legs systemd[1]: kodi.service: Failed with result 'exit-code'.
Sep 21 10:04:19 4legs systemd[1]: Stopped Kodi standalone (X11).

Old thread: https://bbs.archlinux.org/viewtopic.php?id=210674

Last edited by graysky (2020-09-21 15:06:37)

Trilby · 2020-09-21 15:12:18

Building on our email exchange, have you tested whether a manual SIGHUP sent to kodi is sufficient for a clean exit?

Context for everyone else: graysky has reported that manually sending a SIGTERM to kodi is sufficient to get a clean exit. But on termination of the service, systemd would send SIGTERM to xinit (and only to xinit, I believe). Xinit responds to a SIGTERM by sending SIGHUP (instead of SIGTERM) to the client process it started (i.e., Kodi).

graysky · 2020-09-21 15:21:39

1) Start the service
2) As root, sending the SIGHUP at the kodi-x11 PID seems to have no effect. It remains running even after waiting minutes:

# kill -SIGHUP $(pidof kodi-x11)

% ps aux |grep kodi-x11
kodi      121519 68.8  5.7 1606052 223196 tty1   Sl   10:21   0:06 /usr/lib/kodi/kodi-x11 --standalone

By contrast, ending a SIGTERM or TERM kills is within a few seconds and then the systemd service just dies.

Example:

# kill -SIGTERM $(pidof kodi-x11)

... wait a few sec

# systemctl status kodi
● kodi.service - Kodi standalone (X11)
     Loaded: loaded (/usr/lib/systemd/system/kodi.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Mon 2020-09-21 10:24:07 CDT; 11s ago
    Process: 122074 ExecStart=/usr/bin/xinit /usr/bin/kodi-standalone -- :0 -nolisten tcp vt1 (code=exited, status=0/SUCCESS)
   Main PID: 122074 (code=exited, status=0/SUCCESS)

Sep 21 10:23:37 4legs systemd[1]: Started Kodi standalone (X11).
Sep 21 10:23:37 4legs systemd[122074]: pam_systemd_home(login:account): systemd-homed is not available: Unit dbus-org.fre>
Sep 21 10:23:37 4legs systemd[122074]: pam_unix(login:session): session opened for user kodi(uid=989) by (uid=0)
Sep 21 10:24:07 4legs systemd[1]: kodi.service: Succeeded.

Last edited by graysky (2020-09-21 15:25:11)

Trilby · 2020-09-21 15:57:56

So you just need to get xinit (or a replacement) to send SIGTERM. The xinitrc (or just any shell script you'd pass to xinit) approach could be like this:

trap "kill -TERM $(pidof kodi-x11)" SIGHUP
kodi-standalone

So then when xinit gets the SIGTERM from systemd, it will pass on the SIGHUP to xinitrc (or whatever you call this script), which catches this SIGHUP and in turn sends SIGTERM to kodi. Worst case you may need something like this:

trap "kill -TERM $(pidof kodi-x11) && sleep 2" SIGHUP
kodi-standalone

Alternatively, just use a better tool to start X which will actually pass on a SIGTERM to children and wait for them to clean up. EDIT: though I dount my linked implementation will run from a systemd service as it is - it needs to run in a tty - but it could be readily modified to run as a service.

Last edited by Trilby (2020-09-21 15:59:14)

graysky · 2020-09-21 18:32:17

@trilby - Hmmm... I tried your idea for a helper script both with and with a delay (increasing from 2 to 5 s) but the service still ends in a failed state. See: https://github.com/graysky2/kodi-standa … 4fa6e25874

Any thoughts before we move to your alternative plan?

EDIT: how exactly should the compiled xi be used?

xi /usr/bin/kodi-standalone 
get_vt: Inappropriate ioctl for device

Last edited by graysky (2020-09-21 18:39:21)

Trilby · 2020-09-21 19:36:13

Working backwards: that error from xi is a confirmation of my doubt that'd work from a systemd service: it requires an actual tty. This is really only because it sets the vt number and display number for the X Server based on the current tty number. This would be easy to edit if you wanted to use it, but I have no use for a systemd-launched X session, so I've never bothered. Specifically, if you're ok with hardcoding the values:

-	struct vt_stat vtstat;
-	if (ioctl(0, VT_GETSTATE, &vtstat)) {
-		perror("get_vt");
-		return 1;
-	}
-	char dpy[] = { ':', '0' + vtstat.v_active, 0 };
-	char tty[] = { 'v', 't', '0' + vtstat.v_active, 0 };
+	char *dpy = ":1";
+	char *tty = "vt1";

Or you could do a little more modification to take these values from an argument or environment variable. But given the symtpoms, this might all be tangential: does kodi actually end in an unclean state, or is the only issue that the systemd service reports a nonzero exit status?

If kodi exits in an unclean state even with the xinitrc-like helper script, then xi would be worthless too (xi will not accomplish anything that can't be done with the helper script). But if this is really just about the exit code, you could probably just add an `exit` to the end of the helper script, or even just add a last line with just a ":" to the script (while I like the efficiency and elegance of this approach, it'd confuse many readers ... ":" is a shell built in equivalent to "true").

EDIT: the return code might actually have to come from the `trap' scriptlet (what's in quotes as it's first argument). I don't know enough about using `trap` in a shell to really know. I'm pretty sure that control flow would return to the helper script continuing with any lines after kodi-x11 after the signal is trapped and responded to, so a terminal exit/: should work.

Last edited by Trilby (2020-09-21 19:40:00)

graysky · 2020-09-21 20:08:10

I tried adding either the : or exit 0 to the end of the helper script but got the same "failed" status in the systemd service.

#!/bin/bash
trap "kill -TERM $(pidof kodi-x11) && sleep 5s" SIGHUP
/usr/bin/kodi --standalone
exit 0

Hmm... from journalctl:

xinit[6431]: /usr/bin/xinit: connection to X server lost
xinit[6431]:
systemd[1]: Stopping Kodi standalone (X11)...
waiting for X server to shut down
Hangup
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
lircd[615]: lircd-0.10.1[615]: Info: removed client
lircd-0.10.1[615]: Info: removed client
xinit[6452]: (II) Server terminated successfully (0). Closing log file.
xinit[6431]: /usr/bin/xinit: unexpected signal 15
systemd[1]: kodi.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: kodi.service: Failed with result 'exit-code'.
systemd[1]: Stopped Kodi standalone (X11).
systemd[1]: session-45.scope: Succeeded.

I do not know if kodi is actually ending in an unclean state. I assume it is based on the output in the systemctl status message.

Last edited by graysky (2020-09-21 20:13:30)

Trilby · 2020-09-21 21:10:36

Ah, so the issue is that xinit is exiting with an error due to the X server apparently being killed (by getting a SIGTERM from systemd??)

If that's the case, using an ExecStop may work ... though I just don't like systemd's requirement that these are asyncronous processes. If you have an ExecStop that kills kodi (or xinit) that would allow xinit to shutdown the X server before systemd does. A end-kodi script could be:

kill -TERM $(pidof kodi-x11)
sleep 2

And call that from ExecStop. Though I find this pretty ugly - I just don't know if there are good ways to handle this via systemd.

Systemd's greatest strength is that it does everything it can asyncronously. It's greatest weakness is that it does everything it can asyncronously, whether or not it should.

graysky · 2020-09-21 21:25:06

1) I modified /usr/lib/kodi/kodihelper per your suggestion:

#!/bin/bash
kill -TERM $(pidof kodi-x11)
sleep 2

2) I modified /usr/lib/systemd/system/kodi.service as well, adding the following to the [Service] section:

ExecStop=/usr/lib/kodi/kodihelper

After a daemon-reload, the service still ends in a failed state.

Sep 21 17:22:50 myth2 systemd[1]: Stopping Kodi standalone (X11)...
Sep 21 17:22:50 myth2 dbus-daemon[577]: [system] Activating via systemd: service name='org.freedesktop.home1' unit='dbus-org.freedesktop.home1.service' requested by ':1.365' (uid=0 pid=1810 comm="(dih>
Sep 21 17:22:50 myth2 dbus-daemon[577]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.home1.service': Unit dbus-org.freedesktop.home1.service not found.
Sep 21 17:22:50 myth2 systemd[1810]: pam_systemd_home(login:account): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Sep 21 17:22:50 myth2 systemd[1810]: pam_unix(login:session): session opened for user kodi(uid=989) by (uid=0)
Sep 21 17:22:52 myth2 xinit[1711]: /usr/bin/xinit: connection to X server lost
Sep 21 17:22:52 myth2 xinit[1711]: 
Sep 21 17:22:52 myth2 systemd[1814]: pam_unix(login:session): session closed for user kodi
Sep 21 17:22:52 myth2 lircd[607]: lircd-0.10.1[607]: Info: removed client
Sep 21 17:22:52 myth2 lircd-0.10.1[607]: Info: removed client
Sep 21 17:22:53 myth2 xinit[1711]: waiting for X server to shut down
Sep 21 17:22:53 myth2 xinit[1732]: (II) Server terminated successfully (0). Closing log file.
Sep 21 17:22:53 myth2 xinit[1711]: /usr/bin/xinit: unexpected signal 15
Sep 21 17:22:53 myth2 systemd[1]: kodi.service: Main process exited, code=exited, status=1/FAILURE
Sep 21 17:22:53 myth2 systemd[1]: kodi.service: Failed with result 'exit-code'.
Sep 21 17:22:53 myth2 systemd[1]: Stopped Kodi standalone (X11).
Sep 21 17:22:53 myth2 systemd[1]: session-14.scope: Succeeded.

Frustrating, this

loqs · 2020-09-21 22:27:55

What if you specify killmode=process to stop the signals going to the subprocesses. systemd.kill
Edit:
If that fails would it be possible to use PIDFile to set the main process as the X server.

Last edited by loqs (2020-09-21 23:19:36)

graysky · 2020-09-22 12:41:34

@loqs - I added KillMode=Process to the [Service] section but am getting the same failed status on exit. I need to read-up on how the correct usage of PIDFile in this context. Any thoughts before I do that are appreciated. Thanks.

Trilby · 2020-09-22 12:45:33

Did you combine KillMode=Process with the xinitrc wrapper script with the 'trap'?

If I understand the KillMode setting right (which is a long shot), setting it to Process means only the xinit process will get the SIGTERM, then it will be able to pass that on appropriately to the X server and (via the trap) to the kodi process. If systemd's default is to send a SIGTERM to all child processes at the same time, the X Server may die before the xinit process is done with it - which would result in the error you are seeing from xinit.

EDIT: a better approach could be to take xinit out of the equation - and don't bother with a replacement like xi, systemd itself would be the replacement. Run xorg as a service:
https://wiki.archlinux.org/index.php/Sy … er_service
Then run kodi-x11 as a service that depends on the xorg service (requires, after, whatever).

EDIT 2: although this may require a tty login according to the wiki. *shrug*, I can find no rhyme nor reason to system'd black magic. Does this really need to be a service: why not just autologin and autostart X with a simple xinitrc?

Last edited by Trilby (2020-09-22 12:49:52)

graysky · 2020-09-22 15:07:31

@Trilby - I tried pairing KillMode=Process with your wrapper script (with trap) but got the same failed status on exit.

The service is a nice way to have kodi-standalone start for a number of reasons. I will read up on the wiki link you posted to see if having a xorg service could work here.

loqs · 2020-09-22 15:17:14

KillMode=Process sends signal 15 SIGTERM to the main process only xinit. Systemd guessed the wrong main process?

diff --git a/test.c b/test.c
index 74434da..9dbe24e 100644
--- a/test.c
+++ b/test.c
@@ -53,7 +53,9 @@ int client(const char *dpy, const char *wm, const char *xauth) {
 }
 
 int main(int argc, const char **argv) {
+	char *pathname = "/run/xi.pid";
 	struct vt_stat vtstat;
+	FILE *fd;
 	if (ioctl(0, VT_GETSTATE, &vtstat)) {
 		perror("get_vt");
 		return 1;
@@ -76,6 +78,10 @@ int main(int argc, const char **argv) {
 	int cpid = client(dpy, (argc > 1 ? argv[1] : "xterm" ), xauth);
 	while (xrunning) pause();
 
+	fd = fopen(pathname, "w");
+	fprintf(fd, "%i",cpid);
+	fclose(fd);
+
 	kill(cpid, SIGTERM);
 	waitpid(cpid, NULL, 0);
 	kill(xpid, SIGTERM);

In combination with PIDFile=/run/xi.pid

Last edited by loqs (2020-09-22 16:30:08)

graysky · 2020-09-22 19:21:29

I am unclear how to assign the PID manually... systemd service files can indeed be a form of hell.

loqs · 2020-09-22 19:32:09

Is systemd rejecting the .service after you added PIDFile=/run/xi.pid to the [Service] section?

graysky · 2020-09-22 22:53:50

@loqs - It isn't rejecting it, but it doesn't write /run/xi.pid on startup.

[Unit]
Description=Kodi standalone (X11)
After=remote-fs.target systemd-user-sessions.service network-online.target nss-lookup.target sound.target bluetooth.target polkit.service upower.service mysqld.service
Wants=network-online.target polkit.service upower.service
Conflicts=getty@tty1.service

[Service]
User=kodi
Group=kodi
EnvironmentFile=-/etc/conf.d/kodi-standalone
PAMName=login
TTYPath=/dev/tty1
Environment=WINDOWING=x11
ExecStart=/usr/bin/xinit /usr/bin/kodi-standalone -- :0 -nolisten tcp vt1
Restart=on-abort
StandardInput=tty
StandardOutput=journal
PIDFile=/run/xi.pid

[Install]
Alias=display-manager.service

When started, it is running, but there is no /run/xi.pid.

Perhaps since not Type=forking it is silently ignored?

Adding Type=forking breaks it entirely.

Last edited by graysky (2020-09-22 22:56:04)

graysky · 2020-09-23 11:16:41

It seems that adding ExecStop=/usr/bin/pkill kodi is working as expected. Is this sane?

Trilby · 2020-09-23 12:25:45

We already had that before, in a wrapper script and you said it didn't work ... though before it was sending a SIGTERM to kodi-x11, not kodi. I don't know what actual process names kodi uses.

The limitation of a non-wrapper ExecStop is that systemd expects the ExecStop command will not return until after it is complete. Pkill will send the SIGTERM then return. If Kodi ever takes a second to handle the caught SIGTERM there could be problems as systemd will have already killed the server.

graysky · 2020-09-23 12:47:58

@Trilby - Is it a potential race condition you're describing?

For the record, when the service is started there are a number of user kodi owned processes that the service spawns, example:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
kodi        2376  0.0  0.0   3876  1080 tty1     Ss+  08:43   0:00 /usr/bin/xinit /usr/bin/kodi-standalone -- :0 -nolisten tcp vt1
kodi        2385  0.0  0.1  20332 10140 ?        Ss   08:43   0:00 /usr/lib/systemd/systemd --user
kodi        2389  0.0  0.0 180292  3484 ?        S    08:43   0:00 (sd-pam)
kodi        2397  0.0  0.0 180356  3508 tty1     S+   08:43   0:00 (sd-pam)
kodi        2398  0.6  0.4 648696 36652 tty1     Sl   08:43   0:01 /usr/lib/Xorg :0 -nolisten tcp vt1
kodi        2408  0.0  0.0   7268  3528 tty1     S    08:43   0:00 /bin/sh /usr/bin/kodi-standalone
kodi        2410  0.0  0.0   7400  3736 tty1     S    08:43   0:00 /bin/sh /usr/bin/kodi --standalone
kodi        2416  6.7  2.3 2351900 186156 tty1   Sl   08:43   0:17 /usr/lib/kodi/kodi-x11 --standalone

Last edited by graysky (2020-09-23 12:48:44)

Trilby · 2020-09-23 13:18:23

Ah, it's kodi-standalone that would likely really need to get the signal (so pkil kodi works, but not pkill kodi-x11). That may have been the problem with everything based on $(pidof kodi-x11) as the wrong process was being sent the SIGTERM.

And yes, my concern is a race condition. If it works, with just "ExecStop=pkill kodi", then it would work with the wrapper script from earier as long as the SIGTERM is sent to all kodi processes or kodi-standalone rather than just to kodi-x11. But I worry that "ExecStop=pkill kodi" might work most of the time, but "mysteriously" fail on occasion (due to the race going the other way).

However, if the original issue is really just cosmetic, this might be sufficient: if it exits cleanly most of the time, but on occasion due to a race condition does not but this is harmless, then it's probably best to go with the simple approach of "ExecStop=pkill kodi" (which is where this all started).

graysky · 2020-09-23 13:26:59

@Trilby - Yes, it worked years ago with the exception of that /dev/null thing. I have not yet seen that resurfacing so I am assuming it was fixed upstream (systemd). In any case, pushed the ExecStop=pkill kodi to my dev branch. Needs some testing but at least seems to be working. Thanks for all your engagement on this. Thanks to you too, loqs.

graysky · 2020-10-14 12:58:51

@Trilby - What do you think of https://github.com/Earnestly/sx

Trilby · 2020-10-14 13:15:48

In general, or specifically for this?

In general, I like it. I like it so much I was inspired to create 'xi'. It is simple, doesn't do more than it needs to do and it 'traps' and handles relevant signals. Personally I wasn't inclined to use a shell script for this which is why I wrote xi, and while I feel quite proficient in shell scripting, the syntax and flow logic of signal handling in shell scripts has always struck me as a dark magic akin to alchemy: for some reason my brain just doesn't process it well to know where flow control will go and when (there's no logical reason for this as it's really not so different from signal handling in C ... but for some irrational reason, the syntax of it just screws me up).

Perhaps I'm misunderstanding the signal handling in sx, but I don't believe any signals are passed to the child client process: sx traps the relevant signals, but may not pass them to chidlren. In xi, any (handled) signal that causes xi to shutdown results in xi sending SIGTERM to the client/child process - so if xi gets a SIGTERM, it sends a SIGTERM to the client process (and waits for them to finish, which was why I initially suggested it in this thread as other alternatives don't seem to do this). This is in contrast to xinit which (apparently) sends a SIGHUP when it gets a SIGTERM. Initially in this thread, I *thought* that distinction might be relevant. However, it has proven not to be. Sx also handles SIGTERM, but I'm not sure if it then sends anything to the child.

But xi isn't relevant to solving the problem(s) in this thread, and therefore neither is sx. By all means, use xi or sx, they're both great bits of code (I may be biased in assessing one of them), but they aren't relevant to solving the problem in this thread.

Last edited by Trilby (2020-10-14 13:23:13)

graysky · 2020-10-14 13:56:17

OK, thanks for the perspective on it.

Arch Linux

#1 2020-09-21 15:05:49

Tweak a systemd unit for kodi-standalone to allow for a clean exit

#2 2020-09-21 15:12:18

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#3 2020-09-21 15:21:39

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#4 2020-09-21 15:57:56

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#5 2020-09-21 18:32:17

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#6 2020-09-21 19:36:13

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#7 2020-09-21 20:08:10

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#8 2020-09-21 21:10:36

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#9 2020-09-21 21:25:06

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#10 2020-09-21 22:27:55

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#11 2020-09-22 12:41:34

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#12 2020-09-22 12:45:33

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#13 2020-09-22 15:07:31

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#14 2020-09-22 15:17:14

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#15 2020-09-22 19:21:29

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#16 2020-09-22 19:32:09

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#17 2020-09-22 22:53:50

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#18 2020-09-23 11:16:41

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#19 2020-09-23 12:25:45

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#20 2020-09-23 12:47:58

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#21 2020-09-23 13:18:23

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#22 2020-09-23 13:26:59

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#23 2020-10-14 12:58:51

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#24 2020-10-14 13:15:48

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

#25 2020-10-14 13:56:17

Re: Tweak a systemd unit for kodi-standalone to allow for a clean exit

Board footer