[Solved] systemd does not wait for the unit to finish

olive · 2015-03-13 20:04:23

My problem is described in the title. I have made a test by enabling the following unit:

[Unit]
Description=/etc/rc.local Compatibility

[Service]
Type=forking
ExecStart=/bin/sleep 1000
TimeoutSec=0
RemainAfterExit=yes
After=network.target

[Install]
WantedBy=multi-user.target

The unit is well started (a process sleep 1000 exists). But with a type=forking, systemd is supposed to hang in this case. Instead, the graphical.target is launched, systemctl list-units mention it as dead, but the display manager is started before the sleep 1000 completes. Ho can I force systemd to wait?

Last edited by olive (2015-03-16 16:49:34)

Trilby · 2015-03-13 20:12:07

man systemd.service wrote:

Type=
...
If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit ...
Behavior of oneshot is similar to simple; however, it is expected that the process has to exit before systemd starts follow-up units.

Type forking is expected to exit - so there is certainy no assurance that systemd will wait for it to complete. Use oneshot for this.

olive · 2015-03-13 20:26:08

I have tried with oneshot, but this make no difference. The unit is displayed as "activating" and the graphical.taget" is seen as dead. But the the display manager is still launched. I have also tried with variants of the timeout but this makes no difference. I can't make a unit that make systemd to hang.

jasonwryan · 2015-03-13 20:41:20

olive wrote:

I can't make a unit that make systemd to hang.

What are you actually trying to achieve? This sounds like an X-Y problem...

olive · 2015-03-13 21:11:35

I want some operation to complete before the graphical target. for example, I boot the same hard disk in a virtual machine on Windows and natively and I want to make some scripts that adjust some files according to how Archlinux is booted, for example the xorg configuration file. Anyway it seems that there are plenty of reasons we should want for some unit to complete before starting a new one. systemd has "before=" and "after=" directive for that. But it seems that systemd never wait Some my first goal is to understand how can I make systemd to wait for some unit to complete? It seems something very basic.

Last edited by olive (2015-03-13 21:13:19)

Raynman · 2015-03-13 21:16:09

One of the features of systemd is that it starts services in parallel. Your graphical.target wants both multi-user.target and display-manager.service. multi-user.target wants this rc.service (and many other things probably). Unless there are ordering dependencies in place, these services can/will be started in parallel.

However, multi-user.target (and thus also graphical.target) are not finished until the rc.service finishes. That is why graphical.target is listed as "dead" for 1000 seconds. After that, sleep exits and graphical.target becomes active. I tested this with Type=oneshot. forking might work similarly, but it's not really appropriate here (there's no actual forking happening).

Edit (seeing your new reply):

olive wrote wrote:

I want some operation to complete before the graphical target.

I think what you really want is for the operation to complete before display-manager.service, so try adding that with Before=.

Last edited by Raynman (2015-03-13 21:18:07)

olive · 2015-03-14 08:34:46

Raynman wrote:

I think what you really want is for the operation to complete before display-manager.service, so try adding that with Before=.

Here is the graphical target unit:

[Unit]
Description=Graphical Interface
Documentation=man:systemd.special(7)
Requires=multi-user.target
After=multi-user.target
Conflicts=rescue.target
Wants=display-manager.service
AllowIsolate=yes

As you see there is "After=multi-user.target". Of course the display manager is part of the graphical target. So why does systemd does not wait? I have tried to put before=graphical.target in my unit but this make no difference, systemd never wait. Also changing "forking" to "oneshot" does not make a difference. If I put a big sleep in a unit, systemctl list-units reports it as "activating", so it knows that the unit is not yet complete, but nevertheless does not wait whatever the After= or Before= say.

By the way I have a bug that appear occasionally where the screen is black instead of displaying the graphical interface. I am pretty sure it is because the display manager is started too early.

Last edited by olive (2015-03-14 08:55:59)

Raynman · 2015-03-14 09:50:01

If you don't want to actually read my reply or try my suggestion, suit yourself. I learned a little myself so it wasn't a waste of time

You could also try describing precisely why the display manager should not be started until the sleep process finishes (and prove that systemd (and I) ignore(s) Before/After as you claim). I mean, there has to be some path

rc.service -> x1 -> ... -> xN -> display-manager.service

(with N>=0) such that rc.service has Before=x1 or x1 has After=rc.service and so on. Keep in mind that Before/After and Wants/Requires are separate things.

I just tested my suggestion and it works as expected. By the way, (After=) dependencies belong in the [Unit] section.

olive · 2015-03-14 13:08:54

Raynman wrote:

I just tested my suggestion and it works as expected. By the way, (After=) dependencies belong in the [Unit] section.

I have good reasons to want the display manager (and xorg) not to be started before some specific unit completes. In my case it is because I want to change the Xorg.conf file, but we can imagine plenty of other reasons. The graphical target is marked as dead, then as active once the unit completes. However, units that are parts of the graphical target are still launched before the graphical target become active. I am still unable to completely delay the starts of the graphical target before a specific unit completes. It may be normal under some circumstance or according to what I have (mis-)configured, but there must be a way to force systemd to wait until it goes further.

Of course just sleping for 100+ seconds before launching Xorg is stupid in itself, but I put this because it is the simplest example that describes my problem. When something goes wrong (either because of a bug or with a misconfiguration/misunderstanding of my part), I first try to make a minimal reproducible example of the problem.

Last edited by olive (2015-03-14 13:11:30)

berbae · 2015-03-15 16:05:02

olive wrote:

Anyway it seems that there are plenty of reasons we should want for some unit to complete before starting a new one. systemd has "before=" and "after=" directive for that. But it seems that systemd never wait.

The doc is not entire clear on this point, but I seem to understand that the "before=" and "after=" directives configure only a startup ordering of units, ie the order at which units are started, without consideration of their ending times:

unit1 starts -------------------------------------------unit1 ends
             unit2 starts -------------------------------------------unit2 ends
                          unit3 starts -------------------------------------------unit3 ends

unit3 starts after unit2 which starts after unit1.
I may be wrong so if someone knows better than me, please correct me.

There is an example of making systemd to wait, with the 'systemd-networkd-wait-online.service' and the 'systemd-networkd.service', but it is not easy to reproduce, because it uses the 'Type=notify' and apparently a binary which receives the notification messages when the network is ready.

olive wrote:

my first goal is to understand how can I make systemd to wait for some unit to complete? It seems something very basic.

So it doesn't seem so basic.
I will continue to search how this can be done preferably easily.

If someone knows an answer, preferably tested, be nice to post it here, thanks.

teateawhy · 2015-03-15 21:00:39

https://wiki.archlinux.org/index.php/Sy … pendencies

The most typical case is that the unit A requires the unit B to be running before A is started. In that case add Requires=B and After=B to the [Unit] section of A.

Now go read Raynman's last post again.

berbae · 2015-03-15 22:04:56

@teateawhy your reply and Raynman's one work only if the scheme is:

unit1 starts ------------ unit1 ends
                          unit2 starts ---------------- unit2 ends
                                                        unit3 starts ---------------- unit3 ends

and not the one shown in my post #10.
Did you verify that? Can you show how you verified that please?
ie in your example, if B has 'Type=oneshot' and several 'ExecStart=' commands like that:
ExecStart=prog1
ExecStart=prog2
ExecStart=prog3

does A starts only when prog3 exits or just after prog1 starts?
Does A wait or not?

For me the doc is not clear enough, and doesn't seem to tell that systemd waits, instead of just ordering the starts of the dependent units.
But I may be wrong, that's why I would like to see a proof about that, not an ambiguous doc about this point.

Raynman · 2015-03-16 16:10:37

olive, now you're making even less sense. I didn't say the sleep example was stupid and I didn't question your reasons for doing/wanting this.

I suggested you add "Before=display-manager.service" and you respond "I added Before=graphical.target and it didn't change anything."

I also tried to explain why systemd has no reason to delay the display-manager.service. You could have asked for further clarification, as berbae has now done. Let's try a longer explanation.

Service startup

Services can be started in different ways, as configured with Type=. This determines when a service is considered "started" (or when the service's start-up is considered finished). When a service reaches this state (some time after being started), units that are supposed to start After= this service will be started (and no sooner).

With simple systemd has no further information about the start-up process. It launches whatever you specify in ExecStart and this is the main process that continues to run till the service stops. systemd assumes this type of service is started immediately. All the other types have some way for the process to indicate to systemd (either directly or indirectly) when it has finished starting.

Actually oneshot is also a bit special and that is where RemainAfterExit comes in. For oneshot, systemd waits for the process to exit before it starts any follow-up units (and with multiple ExecStarts I assume it waits for all of them). So that automatically leads to the scheme in berbae's last post. However, with RemainAfterExit, the unit remains active even though the process has exited, so this makes it look more like "normal" service with

begin of unit/startup ---- end of startup ------ end of unit

This is the relevant behavior for this thread. First sleep starts, then after 1000 seconds, start-up finishes and follow-up units will be started. Then either the unit dies, or (with RemainAfterExit) it stays "active".

The man page describes how "end of startup" is determined for each Type.

Targets

Targets are meant to group units together, to provide synchronization points (and replace runlevels). When you start a target, all its units will be started (in parallel if possible). The man page says:

Unless DefaultDependencies= is set to false, target units will
implicitly complement all configured dependencies of type Wants=,
Requires=, RequiresOverridable= with dependencies of type After= if the
units in question also have DefaultDependencies=true.

This means that (by default) when a target is requested, all it units are started first. Only after all units have finished starting, the target itself will be started (and since the target doesn't do anything by itself, this startup is basically instantaneous). Without this dependency, the order between the target and its units is unspecified, so in theory the target could finish starting immediately while its units are still being started.

Back to olive

graphical.target has these DefaultDependencies, so it is not started until all its units (like display-manager.service) and other After= dependencies (like multi-user.target) have finished starting. Your sleep service has to be started before multi-user.target starts (again due to default target dependencies). So first display-manager and the sleep service are started and after 1000 seconds, the sleep service finishes starting and then (assuming all other dependencies were quicker) multi-user.target is started and graphical.target as well (assuming display-manager didn't need 1000 seconds).

If you want display-manager.service after the sleep service, add a Before/After line to specify that (this was your original goal and my suggestion).

olive wrote wrote:

However, units that are parts of the graphical target are still launched before the graphical target become active. I am still unable to completely delay the starts of the graphical target before a specific unit completes.

It should be clear now how this works. "units that are part of the graphical target" can only mean "units that are wanted/required by the graphical target" but that is basically all the units that are started when you boot your system, because multi-user.target is a part of graphical.target. And your sleep service is a part of multi-user.target, so in fact you're saying you want to delay starting the sleep service until the sleep service completes

What you probably intended was to delay all units that are a part of graphical.target but not of multi-user.target until after the sleep service. I can't think of an easy (or even good) way to do this and this post is already too long, so I'll table that for now.

olive · 2015-03-16 16:47:33

@Rayman. OK, now I have a working example according to your instructions. Note that I was not saying you were wrong, just that I didn't know how to do want I wanted. I was confused with the different type, the RemainAfterExit and I didn't realize that multi-user.target actually is part of graphical.target (I assumed that it was started afterwards). Anyway, I thank you and I put my working example here:

[Unit]
Description=Test
Before=display-manager.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/bin/sleep 20

[Install]
WantedBy=multi-user.target

Last edited by olive (2015-03-16 16:48:31)

berbae · 2015-03-18 14:55:43

Following my post #12, I made some tests with the case presented there, with the A and B services, with several 'ExecStart=' lines in B.
I can conclude that systemd waits for the end of all the 'ExecStart=' running in B, if there are 'Requires=' and 'After=' in A.
Here is the proof.

1) without the 'After=' dependency

$ cat ~/.config/systemd/user/A.service 
[Unit]
Description=Test systemd waiting
Requires=B.service
#After=B.service

[Service]
Type=oneshot
ExecStart=/usr/bin/echo 'A starts' ; /usr/bin/sleep 10 ; /usr/bin/echo 'A ends'

and

$ cat ~/.config/systemd/user/B.service 
[Unit]
Description=Test oneshot service

[Service]
Type=oneshot
ExecStart=/usr/bin/echo 'prog1 starts' ; /usr/bin/sleep 10 ; /usr/bin/echo 'prog1 ends'
ExecStart=/usr/bin/echo 'prog2 starts' ; /usr/bin/sleep 10 ; /usr/bin/echo 'prog2 ends'
ExecStart=/usr/bin/echo 'prog3 starts' ; /usr/bin/sleep 10 ; /usr/bin/echo 'prog3 ends'

the result is:

$ systemctl --user start A.service
$ journalctl -b -e --user | tail -n 12
mars 18 15:13:59 arch64 systemd[520]: Starting Test systemd waiting...
mars 18 15:13:59 arch64 systemd[520]: Starting Test oneshot service...
mars 18 15:13:59 arch64 echo[3078]: A starts
mars 18 15:13:59 arch64 echo[3079]: prog1 starts
mars 18 15:14:09 arch64 echo[3085]: A ends
mars 18 15:14:09 arch64 systemd[520]: Started Test systemd waiting.
mars 18 15:14:09 arch64 echo[3088]: prog1 ends
mars 18 15:14:09 arch64 echo[3089]: prog2 starts
mars 18 15:14:19 arch64 echo[3097]: prog2 ends
mars 18 15:14:19 arch64 echo[3098]: prog3 starts
mars 18 15:14:29 arch64 echo[3103]: prog3 ends
mars 18 15:14:29 arch64 systemd[520]: Started Test oneshot service.

There A starts at the same time as prog1 in B.

2) With the 'After=B.service' uncommented in A

$ systemctl --user daemon-reload
$ systemctl --user start A.service
$ journalctl -b -e --user | tail -n 13
mars 18 15:16:13 arch64 systemd[520]: Reloading.
mars 18 15:16:16 arch64 systemd[520]: Starting Test oneshot service...
mars 18 15:16:16 arch64 echo[3115]: prog1 starts
mars 18 15:16:26 arch64 echo[3119]: prog1 ends
mars 18 15:16:26 arch64 echo[3120]: prog2 starts
mars 18 15:16:36 arch64 echo[3125]: prog2 ends
mars 18 15:16:36 arch64 echo[3126]: prog3 starts
mars 18 15:16:46 arch64 systemd[520]: Started Test oneshot service.
mars 18 15:16:46 arch64 systemd[520]: Starting Test systemd waiting...
mars 18 15:16:46 arch64 echo[3131]: prog3 ends
mars 18 15:16:46 arch64 echo[3132]: A starts
mars 18 15:16:56 arch64 systemd[520]: Started Test systemd waiting.
mars 18 15:16:56 arch64 echo[3137]: A ends

A starts only after prog3 exits

Arch Linux

#1 2015-03-13 20:04:23