You are not logged in.

#1 2019-11-20 21:50:25

neopium
Member
Registered: 2019-05-02
Posts: 85

[SOLVED] A stop job is running for udev Kernel Device Manager

Hi,

Since the last update, which was quite big with 153 package updates, I have this message displayed in red while the system starts:

A stop job is running for udev Kernel Device Manager

The message is displayed during 20 seconds, then the system boots.

I tried to look at the journalctl messages and I think the problem is around those lines:

Nov 20 23:16:13 archlinux systemd-udevd[361]: Giving up waiting for workers to finish.
Nov 20 23:16:13 archlinux systemd-udevd[361]: Event loop failed: Connection timed out
Nov 20 23:16:13 archlinux systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 23:16:13 archlinux systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Nov 20 23:16:13 archlinux systemd[1]: Stopped udev Kernel Device Manager.
Nov 20 23:16:13 archlinux audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-udevd comm="systemd" exe="/init" hostname=? addr=? terminal=? res=failed'
Nov 20 23:16:13 archlinux systemd[1]: systemd-udevd-control.socket: Succeeded.
Nov 20 23:16:13 archlinux systemd[1]: Closed udev Control Socket.
Nov 20 23:16:13 archlinux systemd[1]: systemd-udevd-kernel.socket: Succeeded.
Nov 20 23:16:13 archlinux systemd[1]: Closed udev Kernel Socket.
Nov 20 23:16:13 archlinux systemd[1]: Starting Cleanup udevd DB...
Nov 20 23:16:13 archlinux systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
Nov 20 23:16:13 archlinux systemd[1]: Stopped Create Static Device Nodes in /dev.
Nov 20 23:16:13 archlinux audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup-dev comm=">
Nov 20 23:16:13 archlinux kernel: audit: type=1131 audit(1574288173.383:11): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=syste>
Nov 20 23:16:13 archlinux kernel: audit: type=1131 audit(1574288173.383:12): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=syste>

Could someone help me fix that slow boot error?

Last edited by neopium (2019-12-04 17:51:08)

Offline

#2 2019-11-20 22:43:54

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

What was in the update?  ( /var/log/pacman.log )
Please post all the message from the start of boot at least up to the entry:

systemd[1]: Stopped Create Static Device Nodes in /dev.

I believe the issue started before the entries you posted.

Offline

#3 2019-11-21 01:18:41

CarbonChauvinist
Member
Registered: 2012-06-16
Posts: 412
Website

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

This actually just happened to me today.

Here's the list of what was updated -- obviously the systemd{,-libs,-sysvcompat} are the prime suspects?:

[2019-11-20T18:00:40-0500] [ALPM] upgraded xdg-utils (1.1.3+18+g0547886-3 -> 1.1.3+18+g0547886-4)
[2019-11-20T18:00:40-0500] [ALPM] upgraded systemd-libs (243.78-2 -> 243.162-2)
[2019-11-20T18:00:40-0500] [ALPM] upgraded systemd (243.78-2 -> 243.162-2)
[2019-11-20T18:00:40-0500] [ALPM] upgraded kirigami2 (5.64.0-1 -> 5.64.1-1)
[2019-11-20T18:00:40-0500] [ALPM] upgraded systemd-sysvcompat (243.78-2 -> 243.162-2)
[2019-11-20T18:00:41-0500] [ALPM] upgraded x264 (3:0.157.r2980.34c06d1-1 -> 3:0.157.r2980.34c06d1-2)
[2019-11-20T19:15:15-0500] [ALPM] upgraded qt5-base (5.13.2-2 -> 5.13.2-3)

Here's a journal

And dmesg as well.

Looks like this is the interesting part?:

[    6.917639] fbcon: Taking over console
[    6.940172] Console: switching to colour frame buffer device 240x67
[   32.060781] kauditd_printk_skb: 4 callbacks suppressed
[   32.060784] audit: type=1131 audit(1574297389.761:15): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-udevd comm="systemd" exe="/init" hostname=? addr=? terminal=? res=failed'
[   32.065188] audit: type=1131 audit(1574297389.767:16): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success'
[   32.065699] audit: type=1131 audit(1574297389.767:17): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=kmod-static-nodes comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success'
[   32.070210] audit: type=1130 audit(1574297389.771:18): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=initrd-udevadm-cleanup-db comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success'
[   32.070213] audit: type=1131 audit(1574297389.771:19): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=initrd-udevadm-cleanup-db comm="systemd" exe="/init" hostname=? addr=? terminal=? res=success'
[   32.115465] systemd-journald[190]: Received SIGTERM from PID 1 (systemd).
[   32.128896] printk: systemd: 18 output lines suppressed due to ratelimiting
[   32.217355] systemd[1]: systemd 243.162-2-arch running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hie
rarchy=hybrid)
[   32.231720] systemd[1]: Detected architecture x86-64.
[   32.232494] systemd[1]: Set hostname to <lap>.
[   32.353478] audit: type=1131 audit(1574297390.054:20): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   32.353693] systemd[1]: initrd-switch-root.service: Succeeded.
[   32.353987] systemd[1]: Stopped Switch Root.
[   32.354142] audit: type=1130 audit(1574297390.054:21): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   32.354158] audit: type=1131 audit(1574297390.054:22): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success

"the wind-blown way, wanna win? don't play"

Offline

#4 2019-11-21 02:01:24

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

https://github.com/systemd/systemd-stab … 243-stable contained a number of commits for udev in the new release (all commits since  Oct 10, 2019)
Not updated this system to systemd 243.162-2.
Adding the boot parameter udev.log_priority=7 may produce more output,  might not work if the init does not use systemd (from the journal looks as though CarbonChauvinist's system does use systemd in init)
Adding the boot parameter systemd.log_level=debug will produce more output from systemd not sure if that includes udev also requires systemd in it.
Adding the boot parameter debug will produce more output but unfortunately also from the kernel and systemd in general.

Offline

#5 2019-11-21 03:19:57

CarbonChauvinist
Member
Registered: 2012-06-16
Posts: 412
Website

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Yes, use systemd init via mkinitcpio for this box.

Tried the "udev.log_priority=7" route. Here's the journal, but not sure if that's more insightful.


"the wind-blown way, wanna win? don't play"

Offline

#6 2019-11-21 06:25:56

neopium
Member
Registered: 2019-05-02
Posts: 85

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

loqs wrote:

What was in the update?  ( /var/log/pacman.log )
Please post all the message from the start of boot at least up to the entry:

systemd[1]: Stopped Create Static Device Nodes in /dev.

I believe the issue started before the entries you posted.

Here is the pacman log extract: https://pastebin.com/QAMgqyt3

It's the big update where they did a massive repackage.

And here is my journalctl log: http://0x0.st/z6dZ.out

Thanks for your help smile

Offline

#7 2019-11-21 14:23:40

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Can you downgrade the systemd packages to confirm that they are the cause?
If so if you rebuild systemd locally in a clean chroot but change the commit to ce4d17ddbc16df05c8dd9f69c65c2570f7b3b374 ,
which is just before most of the udev changes is the issue still present?

Offline

#8 2019-11-21 14:41:47

Ropid
Member
Registered: 2015-03-09
Posts: 1,069

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

I see the same problem here when using the "systemd" hook for the initramfs. There's a delay of 30 seconds between the first log messages inside the initramfs environment and systemd's "Reached target Switch Root" message. I tried using to the "udev" hook instead in /etc/mkinitcpio.conf, and the 30 sec delay is not happening with that. The two hooks are using the same systemd-udevd binary, so maybe the bug is in systemd and not in systemd-udevd?

Offline

#9 2019-11-21 18:48:10

r3b311i0n
Member
From: Sri Lanka
Registered: 2016-04-06
Posts: 26
Website

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Getting the same issue. Downgrading to systemd-243.78-2 removes the delay.

Offline

#10 2019-11-21 19:55:10

neopium
Member
Registered: 2019-05-02
Posts: 85

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

r3b311i0n wrote:

Getting the same issue. Downgrading to systemd-243.78-2 removes the delay.

This solves it for me too.

Don't know if I should mark the thread as SOLVED yet though... This is just a workaround

Offline

#11 2019-11-21 20:21:06

neopium
Member
Registered: 2019-05-02
Posts: 85

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

loqs wrote:

Can you downgrade the systemd packages to confirm that they are the cause?
If so if you rebuild systemd locally in a clean chroot but change the commit to ce4d17ddbc16df05c8dd9f69c65c2570f7b3b374 ,
which is just before most of the udev changes is the issue still present?

Rebuilding systemd in a clean chroot is above my current linux level I fear smile

I'm not sure where I would start.

From my understanding (I cheated and read the wiki page too), chroot makes the system think that the root directory is in a different place...

But what should I actually do?

I would be thinking about something like:

$ mkdir tmp
$ sudo chroot tmp

What then?

$ git clone https://github.com/systemd/systemd-stable.git
$ cd systemd-stable.git
$ git reset --hard ce4d17ddbc16df05c8dd9f69c65c2570f7b3b374

Then make, make install?

Sorry, I haven't compiled C code for years big_smile

Last edited by neopium (2019-11-21 20:22:35)

Offline

#12 2019-11-22 07:47:14

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,427

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

No, to properly and following the Arch Way™ do this you should follow: https://wiki.archlinux.org/index.php/De … ean_chroot

Offline

#13 2019-11-22 09:22:04

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Install the devtools package then:

git clone git://git.archlinux.org/svntogit/packages.git --single-branch --branch "packages/systemd
cd packages/trunk

Edit the PKGBUILD changing
_commit='e51d9bf9e5ac5a6618c175cd9b5cfdc6733cd5d1' to _commit='ce4d17ddbc16df05c8dd9f69c65c2570f7b3b374'
GnuPG#Use_a_keyserver to import the PGP keys listed in PKGBUILD.

extra-x86_64-build

Then install the packages with pacman -U.

Last edited by loqs (2019-11-22 09:29:43)

Offline

#14 2019-11-22 15:06:10

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Updated to systemd 243.162-2 with one irritating issue.

Using the scripts/hook from the Dracut wiki (WIP) page.

I use dracut for my normal initramfs generation, but still keep mkinitcpio as a standby until the 'official' dracut hooks are published.  Filenames do not overwrite each other:

Dracut:  dinitramfs-linux.img
mkinitcpio:  initramfs-linux.img

Anyway, after systemd upgrade and reboot (with dracut initramfs), I have an annoying message, that lasts for about 20 seconds:

A stop job is running for udev Kernel Device Manager

Journal:

journalctl -b -1 -p3
-- Logs begin at Fri 2019-09-27 09:14:11 EDT, end at Fri 2019-11-22 09:43:19 EST. --
Nov 22 09:25:55 Jammin1 systemd-udevd[291]: Giving up waiting for workers to finish.
Nov 22 09:25:55 Jammin1 systemd-udevd[291]: Event loop failed: Connection timed out
journalctl -b -1 | grep systemd-udevd
Nov 22 09:25:24 Jammin1 systemd-udevd[291]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Nov 22 09:25:55 Jammin1 systemd-udevd[291]: Giving up waiting for workers to finish.
Nov 22 09:25:55 Jammin1 systemd-udevd[291]: Event loop failed: Connection timed out
Nov 22 09:25:55 Jammin1 systemd[1]: systemd-udevd.service: Main process exited, code=exited, status=1/FAILURE
Nov 22 09:25:55 Jammin1 systemd[1]: systemd-udevd.service: Failed with result 'exit-code'.
Nov 22 09:25:55 Jammin1 systemd[1]: systemd-udevd-control.socket: Succeeded.
Nov 22 09:25:55 Jammin1 systemd[1]: systemd-udevd-kernel.socket: Succeeded.
Nov 22 09:25:55 Jammin1 systemd-udevd[436]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Nov 22 09:25:55 Jammin1 systemd-udevd[436]: /usr/lib/udev/rules.d/11-dm-lvm.rules:40 Invalid value for OPTIONS key, ignoring: 'event_timeout=180'
Nov 22 09:25:55 Jammin1 systemd-udevd[436]: /usr/lib/udev/rules.d/11-dm-lvm.rules:40 The line takes no effect, ignoring.
Nov 22 09:25:56 Jammin1 systemd-udevd[465]: Using default interface naming scheme 'v243'.

Also, I don't use LVM...

None of the above is present (including the stop job) if booting with mkinitcpio initramfs:

journalctl -b 0 -p3
-- Logs begin at Fri 2019-09-27 09:14:11 EDT, end at Fri 2019-11-22 09:43:19 EST. --
-- No entries --

Any ideas on where to start troubleshooting?

Updating:

When adding 'systemd' to mkinitcpio.conf and regenerating/rebooting, the same 'stop job' delay appears.

Last edited by merlock (2019-11-22 19:52:51)


Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#15 2019-11-22 15:22:18

CarbonChauvinist
Member
Registered: 2012-06-16
Posts: 412
Website

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

This is being discussed in the unfortunately titled thread here.

Are you using systemd init or busybox init for your mkinitcpio initramfs?

I've found, in my cursory tests at least, that busy box is not affected, additionally there may be something somewhat non-deterministic about the error with the systemd init as I have had two boots with systemdinit where I didn't get the delay. In both those boots I'd just changed the kernel command line parameters in my systemd-boot conf files to add some options related to trying to capture logs in the initramfs to see what the delay was - however on subsequent reboot with the same parameters I got the delay.

Last edited by CarbonChauvinist (2019-11-22 15:22:56)


"the wind-blown way, wanna win? don't play"

Offline

#16 2019-11-22 16:12:39

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Merge requested.


Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#17 2019-11-22 17:51:11

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,739

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Merged, by request.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#18 2019-11-22 18:18:11

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

CarbonChauvinist wrote:

Are you using systemd init or busybox init for your mkinitcpio initramfs?

I'm not sure what you mean;  I'm not using busybox...

These are my 'hooks' from /etc/mkinitcpio.conf:

HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)

Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#19 2019-11-22 18:33:15

loqs
Member
Registered: 2014-03-06
Posts: 17,196

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

If you do not add the systemd hook then init in the initrd is a shell script executed by busybox.

Offline

#20 2019-11-22 19:19:51

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

loqs wrote:

If you do not add the systemd hook then init in the initrd is a shell script executed by busybox.

Learned something new.  Thanks, @loqs.


Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#21 2019-11-23 09:38:17

Master One
Member
From: Europe
Registered: 2007-01-21
Posts: 249

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Ropid wrote:

I see the same problem here when using the "systemd" hook for the initramfs. There's a delay of 30 seconds

Problem confirmed, the same thing is happening here since the recent systemd upgrade. The message "A stop job is running for udev Kernel Device Manager" stays for 30 seconds, then boot continues without (visible) errors. I can not see a problem afterwards, so the only issue is the 30 seconds delay during boot (which doesn't bother me, I only reboot after the daily system upgrade), so I guess I can leave it as it is till a s solution is found.

neopium wrote:

Don't know if I should mark the thread as SOLVED yet though... This is just a workaround

A systemd downgrade as a workaround is no solution.

Offline

#22 2019-11-23 09:40:07

Master One
Member
From: Europe
Registered: 2007-01-21
Posts: 249

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Maybe the OP or a mod can change the thread title to be more specific?

Last edited by Master One (2019-11-23 09:41:04)

Offline

#23 2019-11-23 11:18:38

neopium
Member
Registered: 2019-05-02
Posts: 85

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Master One wrote:

Maybe the OP or a mod can change the thread title to be more specific?

I can do that, yes.

The thing is, I don't follow everything about those hooks...

I suggest: Systemd udev hangs at startup

Would this thread title be better?

Offline

#24 2019-11-23 11:23:21

Master One
Member
From: Europe
Registered: 2007-01-21
Posts: 249

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

neopium wrote:

I can do that, yes. The thing is, I don't follow everything about those hooks... I suggest: Systemd udev hangs at startup. Would this thread title be better?

I'd just use the error message ("A stop job is running for udev Kernel Device Manager") as thread title, because that's what I used as the search phrase. wink

Last edited by Master One (2019-11-23 11:23:43)

Offline

#25 2019-11-23 11:28:51

neopium
Member
Registered: 2019-05-02
Posts: 85

Re: [SOLVED] A stop job is running for udev Kernel Device Manager

Master One wrote:
neopium wrote:

I can do that, yes. The thing is, I don't follow everything about those hooks... I suggest: Systemd udev hangs at startup. Would this thread title be better?

I'd just use the error message ("A stop job is running for udev Kernel Device Manager") as thread title, because that's what I used as the search phrase. wink

Done

Offline

Board footer

Powered by FluxBB