controlling systemd shutdown order

anne · 2017-02-09 22:00:46

Please save my sanity; I've been reading manpages and articles for
hours without finding a successful solution. This may be because
I'm trying to do something which sort of violates the spirit of
systemd - but it should still be possible!!!

When the system starts to shut down (or reboot), I'd like it to
run a script, and I'd like systemd to wait for my script to finish,
*before* systemd starts to kill user processes.

I'm able to define a service (Type=oneshot, RemainAfterExit=yes)
and enable it, and my ExecStop is duly called during shutdown, but
too late. Since the shutdown order is supposed to be the reverse
of the start-up order, I would have thought that I could come up
with some appropriate "After=" specifications that would cause my
service to be stopped before the "Stopping Session" tasks started
to kick in, but so far, no go. I've even tried:

Requires=systemd-user-sessions.service systemd-logind.service
After=systemd-user-sessions.service systemd-logind.service

but it looks as though systemd pounces to start killing user sessions
very early, no matter what I do. My ExecStop is always called too late.

Obviously I could alias or replace the "shutdown" and "reboot"
commands, but that would be a dirty hack.

Any ideas on how to do this properly within systemd?

Anne.

anne · 2017-02-18 03:16:47

Over a week and zero replies! Are there no systemd experts around at the moment?

There must be a way to convince systemd to run a script at shutdown, and wait for it to finish before starting to kill user processes. What am I missing?

Anne.

Docbroke · 2017-02-18 03:38:32

anne wrote:

When the system starts to shut down (or reboot), I'd like it to
run a script, and I'd like systemd to wait for my script to finish,
*before* systemd starts to kill user processes.
...

Once the shutdown process starts you cannot run your script. Only way is to run your script before calling the process of shutdown.

anne wrote:

...
Obviously I could alias or replace the "shutdown" and "reboot"
commands, but that would be a dirty hack.
...

I think that is what exactly you are asking for, "replacing the systemd shutdown/reboot with your own process"

AND.. bumping the thread is not received well on this forums.

EDIT: spell correction from will to well

Last edited by Docbroke (2017-02-18 03:52:25)

brebs · 2017-02-18 07:54:09

anne wrote:

When the system starts to shut down (or reboot), I'd like it to run a script

What triggers that shutdown? You could write a custom shutdown script, which does the stuff you want, and then triggers a normal shutdown.

If you provide some info on the script's purpose, we might be able to help solve the real problem.

anne · 2017-02-18 16:13:15

Thank you very much, Docbroke and Brebs, for taking the time to reply.

Yes, of course I could replace the shutdown and reboot scripts; that
would be trivial. However, it would be a hack, and I'm trying to
learn how to write systemd units correctly.

Docbroke wrote:

Once the shutdown process starts you cannot run your script. Only
way is to run your script before calling the process of shutdown.

That's not the case at all - in fact, my script does run.
It just runs too late. According to the manpage for "systemd.unit":

when two units with an ordering dependency between them are shut
down, the inverse of the start-up order is applied. i.e. if a
unit is configured with After= on another unit, the former is
stopped before the latter if both are shut down.

So it should be possible to ensure that my service is shut down
(and therefore, my script as specified via ExecStop runs) before
another service gets shut down, by making my script "want" or
"require" that other service, and start "after" it.

What I need to figure out (I think!) is: what is the service that
is responsible for user processes, and therefore that kills user
processes on shutdown? Then I could make my service start after
that, and thus be shut down before it.

Alternatively, if user processes are killed not by shutting down a
parent service of theirs, but by starting some other service that
itself is required/wanted by the shutdown.target, then what is this
service that kills the user processes?

If there's documentation out there that describes the systemd shutdown
process in more detail than the manpages, I haven't managed to find
it yet. I'm happy to read the docs and the various unit definitions
to figure this out, but I'm stuck as to where to look.

Anne.

brebs · 2017-02-18 16:29:43

Just in case you don't get a better answer, there is always the source code to peruse.

Trilby · 2017-02-18 16:30:40

brebs wrote:

If you provide some info on the script's purpose, we might be able to help solve the real problem.

At least let us know how long this script is likely needed to complete. Might it be more than 90 seconds? If so, have you set TimeoutStopSec? or DefaultTimeoutStopSec?

berbae · 2017-02-18 16:31:15

You could add an 'ExecStop=' line for the service for which you want the script to run before it stops, using a drop-in conf file for this service.

anne · 2017-02-18 17:24:21

Trilby wrote:

At least let us know how long this script is likely needed to complete. Might it be more than 90 seconds? If so, have you set TimeoutStopSec? or DefaultTimeoutStopSec?

Not likely more than 5 seconds, but you're barking up the wrong tree.
It starts and it completes, but by the time it starts, the system has
already started killing user processes, and I want this script to run
before any end-user processes are killed.

What I can't seem to figure out is how to set up a dependency to
accomplish this.

Anne.

anne · 2017-02-18 17:33:22

berbae wrote:

You could add an 'ExecStop=' line for the service for which you want the script to run before it stops, using a drop-in conf file for this service.

I'd be happy to do this, if I could figure out which service is
killing the end-user processes during shutdown. ... but if I
could figure that out, I could probably write the dependency!

(That is, if "Type=oneshot" successfully makes follow-up services
wait for its completion before starting their own shutdown - the
manpage explains that "the process has to exit before systemd starts
follow-up units" during start, but it's not clear that this
restriction is also enforced during shutdown.)

At the moment, though, I can't even make my "service" start shutting
down first, so making sure that any other service waits for it to
finish shutting down is rather moot.

Anne.

Trilby · 2017-02-18 20:03:32

anne wrote:

... but you're barking up the wrong tree.

Yeah, I guess you fooled me. Not providing relevant information, and not sharing the purpose of the script after two requests will have that effect. Good luck - I'm not sure anyone will care to try to help you if you persist like this.

anne · 2017-02-18 22:21:37

Trilby wrote:

anne wrote:
... but you're barking up the wrong tree.
Yeah, I guess you fooled me. Not providing relevant information, and not sharing the purpose of the script after two requests will have that effect. Good luck - I'm not sure anyone will care to try to help you if you persist like this.

I'm sorry, I'm not trying to be mysterious or difficult, but I
am trying to understand systemd; to some extent this is an
exercise for me for that purpose. I'm eventually going to need
to make sure that production systems get their services shut down
cleanly: apps before databases, etc. This seemed as though it
would be straightforward, based on the manpages, and quite possibly
it will be, and perhaps I'm stuck now only because the "service"
I'm trying to depend on isn't really a service - though I thought
that most things were services.

The actual problem I'm trying to solve with this is not that
important in itself, but, since you folks really want to know:

I start my X session manually with "startx", and I'd like it to
close down gracefully, with certain applications (xterms) being
killed first, with a short sleep between each kill. The original
reason for this was that I would get hangs during system shutdown
and even during first login after a reboot, which I eventually traced
to contention by tcsh over the .history.lock file. That particular
problem was eliminated for logging in by adding code to .cshrc to
blow that file away if it persists for more than one second, to
avoid .login blocking indefinitely trying to obtain this lockfile.
I also reduced the DefaultTimeoutStopSec to reduce the pain when this
happens on shutdown. Finally, if I exit X gracefully, all's well: my
xinitrc now has a script that runs right after the preferred client
exits, which kills the xterms one by one to reduce the chances of
tcsh tripping over itself with the lock files. (It also looks cool to
have the windows go away one by one, but that's neither here nor there!)

Of course, all of this suggests a bug in tcsh whereby file locking is
not done correctly, but it's beyond my ability in the short term to
track that down, and my workarounds work well enough. If I managed
desktops in production, I'd have to dig into why tcsh doesn't clean
up its .history.lock file properly, but fortunately, in production
I manage only servers, where I'm unlikely to encounter this problem. ;-)

In any case, my original problem of tcsh hangs on my home desktop has
been worked around as long as I exit X before shutting down the system.

However, while working on that problem, I became determined to learn
how to make systemd run a script before killing user processes,
with the intention of having a script kill my preferred client
and wait for the X server to disappear. I have a script that
does exactly that, but no matter what I do, I can't seem to make
systemd's killing of the user processes wait for my script to run.

I defined a service and tried various dependency and ordering
statements to try to convince systemd to stop my service before
killing the user processes. "systemctl status" shows my Xorg and
xterms and other X apps, as well as my shells, as children of
"session-c2.scope", which is in turn a child of "user-1043.slice",
which is a child of "user.slice". So in desperation I threw
everything at it and tried:

Requires=systemd-user-sessions.service systemd-logind.service user.slice local-fs.target
After=systemd-user-sessions.service systemd-logind.service user.slice local-fs.target

... hoping that my service would be stopped before the user slice was
stopped. No go.

I also tried alternative approaches where I made a service that was
wanted by and to be started before the shutdown.target, where the
start of the service would have been intended to run my script.
(By this point, I was simply creating a bunch of bogus units that
called "/usr/bin/logger" for ExecStart, ExecStop, ExecStop, and
ExecStopPost, just to see what was happening.) In pretty much
all cases, the starts happened at the right moment, and the stops
happened later than I wanted.

Finally, I really do appreciate efforts to help; people's time is
valuable and I don't want to waste it. But with all due respect,
I did provide relevant information in my original posting.
I explained that I was able to define a service and my ExecStop
was duly called during shutdown, but too late. I asked for help in
determining a dependency that would cause my service to be stopped
before the "Stopping Session" tasks started to kick in.

I assumed that I was missing some simple piece of knowledge with
respect to what triggers the user process kills during shutdown, and
that if I had this piece of knowledge, I'd be able to write a unit
file to call my script at the right moment.

The reason why I want to call a script before user processes are
killed isn't all that important: I might be taking system diagnostic
snapshots, for example, or automatically sending e-mail to everyone
logged in, to apologize for an unannounced reboot. No matter what
arbitrary reason I have, it should still be possible to do this, and
I'm quite astonished that it has turned out to be so difficult.

If someone knows what triggers the user session stops, and how to
depend on it, I hope that they will share that knowledge.

Thanks to all who've piped up, and I apologize for annoying some of you.

Anne.

brebs · 2017-02-18 22:57:20

Hmm, looking at the code, searching for "poweroff", leads to poweroff, and then shutdown.

Perhaps one could use shutdown.target to run a script.

damjan · 2017-02-19 22:58:52

check the boot order when booted, and place your service after the last thing started. Maybe after user@1000.service?

systemd-user-sessions.service is not what you are looking for, check the man page.

Maniaxx · 2017-02-21 02:20:11

ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot) so other units might kick in earlier.
Even though 'After=multi-user.target' starts the unit very early in the shutdown process already the script invocation itself (*.sh) might need too much time so other (parallel started) units might come in too fast. Thus the 'sleep 1' that keeps them away (oddly, this is not parallelized. The 'sleep' properly blocks the whole systemd).
Just a dirty hack but i haven't found anything proper either.

Edit: Correction. Its not reliable. 'After=multi-user.target' still starts it too late sometimes.

[Unit]
Description=Unmount personal blocking mounts
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/etc/early-shutdown-script.sh
ExecStop=/usr/bin/sleep 1

[Install]
WantedBy=multi-user.target

Last edited by Maniaxx (2017-02-22 00:48:45)

anne · 2017-03-08 02:46:05

Folks, sorry about the long silence; I was on vacation.

Brebs, I had already tried using shutdown.target to start the script, but it still starts too late.

Damjan, I was able to start the service very late in the boot sequence, but still couldn't get it to start shutting down early enough.

Maniaxx, my experience mirrors yours, that even "After=multi-user.target" isn't sufficient to make the shutown start first. I'm glad that you're also finding that "ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot)", and I really wonder whether or not "oneshot" should wait on shutdown; as I recall from studying the manpage, no explicit statement is made one way or the other. I might try to find a spot to report that omission; perhaps the systemd people can at least clarify what's supposed to happen there. I'm intrigued by the idea that the "sleep" blocks everything, though - I'll experiment with that in case it gives me useful ideas.

I probably won't work on this again for the next few weeks, but if I get anything to work, I'll report back.

Thanks to all for your help,

Anne.

intelfx · 2017-04-08 13:46:02

Maniaxx wrote:

ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot) so other units might kick in earlier.

anne wrote:

Maniaxx, my experience mirrors yours, that even "After=multi-user.target" isn't sufficient to make the shutown start first. I'm glad that you're also finding that "ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot)", and I really wonder whether or not "oneshot" should wait on shutdown; as I recall from studying the manpage, no explicit statement is made one way or the other. I might try to find a spot to report that omission; perhaps the systemd people can at least clarify what's supposed to happen there.

Uhhuh. Surely it does (per the ordering dependencies). It's just specifying After=multi-user.target in a unit WantedBy=multi-user.target is creating an ordering dependency loop (because if a target Wants= a unit, then that unit is implicitly Before= that target — unless you say DefaultDependencies=no), and systemd has to throw out one of the conflicting dependencies. Unfortunately, it does not have any concept of "priority" to prefer throwing out an implicit dependency — so it may very well prefer the implicit one and discard what you wrote explicitly.

Now replying to the original thread. Anne, you seem to want to run your unit before the session scopes are killed, correct? This, I'm afraid, is impossible with current systemd: session scope units are autogenerated, and there is no way to specify an After= dependency on all of them.

However, the dependency mechanism per se is not flawed in any way, it works, and if you magically knew the name of the needed session scope, then you could simply write After= to that scope (and After=user@<uid>.service to work around an unrelated bug which is going to be fixed before next release when LP reviews my pull requests), and your oneshot unit will be stopped before systemd even attempts to kill that session.

You can try to propose a RFE to make systemd add an "include" statement to the autogenerated session scope units, so that you can put your own snippets with ordering dependencies into that file and have them included in all autogenerated session scope units.

Last edited by intelfx (2017-04-08 13:47:36)

damjan · 2017-04-13 09:49:56

intelfx wrote:

Uhhuh. Surely it does (per the ordering dependencies). It's just specifying After=multi-user.target in a unit WantedBy=multi-user.target is creating an ordering dependency loop (because if a target Wants= a unit, then that unit is implicitly Before= that target — unless you say DefaultDependencies=no), and systemd has to throw out one of the conflicting dependencies.

are you sure about this? the systemd.target man page says:

AUTOMATIC DEPENDENCIES
       Unless DefaultDependencies= is set to no in either of related units or an explicit ordering dependency is already defined,
       target units will implicitly complement all configured dependencies of type Wants= or Requires= with dependencies of type
       After=. Note that Wants= or Requires= must be defined in the target unit itself — if you for example define Wants=some.target
       in some.service, the implicit ordering will not be added.

galexand · 2017-04-30 02:11:44

i think anne's request is very reasonable, which is to say i had the same desire and i am still dismayed that systemd has the attitude it has towards processes. in essence, it demands that every single process be part of its grand plan for the universe or it kills it off immediately. Yuck.

anyways, one thing I found that at least shines a little light on that is "systemctl status <pid>", which shows you which unit it thinks the process belongs to. so if you want to make sure your unit's ExecStop runs before a specific process is killed, you can at least find out which unit you have to depend on.

but i think madness is at the end of that road - systemd really doesn't want to admit there's a value to processes it doesn't explicitly know. so i counsel letting systemd know the exact dependency by putting the process that you care about inside of a unit. i don't like it, but it is the systemd way, and for as stubborn as systemd is, it is not hard to make a new unit. so instead of starting your process directly, you would wrap it in foo.service and do "systemctl start foo.service" to start it, or something like that.

assuming it can run without the terminal, right?

that's easy to do if you're running everything as root. if you need to depend on a non-root process, i'm not entirely clear on that, but systemd.exec man page's User= looks promising? supposedly polkit can let you use systemctl as a user..

hope this helps

galexand · 2017-04-30 02:14:41

oh and as an aside, once i finally got the Wants=/Requires= and After= lines down how i needed them, i found systemd totally reliable about waiting until the Type=oneshot ExecStop= command has finished before stopping the next unit. so i think once you find the right thing to depend upon, it really will wait until your script is finished.

meleu · 2017-09-17 13:04:00

I'm highly interested in solve this. I have a very similar scenario, like the OP.

To be honest the system I'm using is Raspbian, on a Raspberry Pi. But it's related to the systemd question here, and I think it applies too.

What I want to do is to execute a custom script right before the system goes down, and that the shutdown/reboot procedure wait my custom script finish before starting killing processes.

If you are curious, the custom script is this one: https://ghostbin.com/paste/owbqn

What it does is softly kill an application named emulationstation, and therefore saving some data related with this application.

What I've reached so far:

- created that script and named it as "/home/pi/bin/killes.sh".

- created this unit

[Unit]
Description=Kill EmulationStation
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=-/home/pi/bin/killes.sh

[Install]
WantedBy=shutdown.target reboot.target halt.target

I've made some tests changing the script content with "touch /delete-me" and I'm sure the script is running. But when I need a little more elaborated script it doesn't work as expected.

Thanks in advance.

fsckd · 2017-09-17 13:39:07

meleu, have you seen our Code of conduct, particularly this section called Arch Linux distribution support *only*? We are particularly strict about it.

Also, this thread is getting on in age and since OP hasn't followed up to the later replies, I'll close this as per our policy.

Arch Linux

#1 2017-02-09 22:00:46

controlling systemd shutdown order

#2 2017-02-18 03:16:47

Re: controlling systemd shutdown order

#3 2017-02-18 03:38:32

Re: controlling systemd shutdown order

#4 2017-02-18 07:54:09

Re: controlling systemd shutdown order

#5 2017-02-18 16:13:15

Re: controlling systemd shutdown order

#6 2017-02-18 16:29:43

Re: controlling systemd shutdown order

#7 2017-02-18 16:30:40

Re: controlling systemd shutdown order

#8 2017-02-18 16:31:15

Re: controlling systemd shutdown order

#9 2017-02-18 17:24:21

Re: controlling systemd shutdown order

#10 2017-02-18 17:33:22

Re: controlling systemd shutdown order

#11 2017-02-18 20:03:32

Re: controlling systemd shutdown order

#12 2017-02-18 22:21:37

Re: controlling systemd shutdown order

#13 2017-02-18 22:57:20

Re: controlling systemd shutdown order

#14 2017-02-19 22:58:52

Re: controlling systemd shutdown order

#15 2017-02-21 02:20:11

Re: controlling systemd shutdown order

#16 2017-03-08 02:46:05

Re: controlling systemd shutdown order

#17 2017-04-08 13:46:02

Re: controlling systemd shutdown order

#18 2017-04-13 09:49:56

Re: controlling systemd shutdown order

#19 2017-04-30 02:11:44

Re: controlling systemd shutdown order

#20 2017-04-30 02:14:41

Re: controlling systemd shutdown order

#21 2017-09-17 13:04:00

Re: controlling systemd shutdown order

#22 2017-09-17 13:39:07

Re: controlling systemd shutdown order

Board footer