Failed at step EXEC spawning /bin/plymouth: No such file or directory

bigblondewolf · 2013-09-17 22:24:47

Hi guys, Yet Another Plymouth Problem:

I just did a system update. Haven't done this for several months and problems with /bin happened, as expected. Anyway, after update completed, the system won't boot normally:

If I type ^d boot continues with no problems and all services are started as expected. I have trouble finding what causes the system to go to emergency mode. The /srv/nfs/trunk/ seems to be mounted at a second attempt.

An excerpt from journalctl -xb, right after the xfs messages :

Sep 18 00:55:50 xxxlinux systemd[1]: Startup finished in 2.322s (kernel) + 1.350s (userspace) = 3.672s.
-- Subject: System start-up is now complete
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- All system services necessary queued for starting at boot have been
-- successfully started. Note that this does not mean that the machine is
-- now idle as services might still be busy with completing start-up.
--
-- Kernel start-up required 2322291 microseconds.
--
-- Initial RAM disk start-up required INITRD_USEC microseconds.
--
-- Userspace start-up required 1350682 microseconds.
Sep 18 00:55:50 xxxlinux systemd[168]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
-- Subject: Process /bin/plymouth could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/641257651c1b4ec9a8624d7a40a9e1e7
--
-- The process /bin/plymouth could not be executed and failed.
--
-- The error number returned while executing this process is 2.
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Version 2013.1.13 external FUSE 29
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Mounted /dev/sdb6 (Read-Write, label "", NTFS 3.1)
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Cmdline options: rw,uid=1000,gid=100,dmask=022,fmask=133
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Mount options: rw,allow_other,nonempty,relatime,default_permissions,fsname=/dev/sdb6,blkdev,blk
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Global ownership and permissions enforced, configuration type 7
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:0: Direct-Access     Generic  USB SD Reader    1.00 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:1: Direct-Access     Generic  USB CF Reader    1.01 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:0: [sde] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:2: Direct-Access     Generic  USB SM Reader    1.02 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:1: [sdf] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:3: Direct-Access     Generic  USB MS Reader    1.03 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:2: [sdg] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:3: [sdh] Attached SCSI removable disk

After I press ^d:

Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Sound Card.
-- Subject: Unit sound.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sound.target has begun shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopped target Sound Card.
-- Subject: Unit sound.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/9d1aaa27d60140bd96365438aad20286
--
-- Unit sound.target has finished shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Emergency Mode.
-- Subject: Unit emergency.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit emergency.target has begun shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopped target Emergency Mode.
-- Subject: Unit emergency.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/9d1aaa27d60140bd96365438aad20286
--
-- Unit emergency.target has finished shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Emergency Shell...
-- Subject: Unit emergency.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit emergency.service has begun shutting down.

And the system boots.

I found a thread explaining that there is a sysV runlevel specified in kernel boot line. I am not sure if I am looking at the proper file, but my /boot/grub/grub.cfg does not contain "runlevel".
I also found it may be a problem of software RAID, but I am not using it neither LVM, plain mounts by UUID.
I ran pacman -S linux, /etc/mkinitcpio.conf does not contain a reference to plymouth: HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

Just checked with my laptop, both computers run systemd-204-3, neither have /bin/plymouth, but the laptop does not fail with this message

Just tell me if you need more information.

cfr · 2013-09-19 01:03:37

What is your kernel command line? If it ends with a number such as "3" or "5", that's a runlevel and you need to change it or get rid of it. If it is "5", you can just delete it - systemd will default to that anyway. If it is "3" replace it with systemd.unit=multi-user.target.

EDIT: I thought this change came in systemd 207. At any rate, if you are not using 207, you should update.

Last edited by cfr (2013-09-19 01:04:48)

bigblondewolf · 2013-09-19 07:37:42

cfr wrote:

What is your kernel command line? If it ends with a number such as "3" or "5", that's a runlevel and you need to change it or get rid of it. If it is "5", you can just delete it - systemd will default to that anyway. If it is "3" replace it with systemd.unit=multi-user.target.

Should this be in grub.cfg? The only thing that looks like a kernel command line there is within the menuentry block:

linux   /vmlinuz-linux root=UUID=775642a3-f485-4cc6-b9d5-c336aa276c39 rw

cfr wrote:

EDIT: I thought this change came in systemd 207. At any rate, if you are not using 207, you should update.

I thought I did a full system update (pacman -Syu) on Monday, systemd was still 204. I did an update again, now at 207 the problem persists.

cfr · 2013-09-19 21:40:41

Did you execute the command systemd suggests to get information about why srv-nfs-trunk.mount is failing?

The reason I asked about runlevels was because the plymouth error can indicate that problem and is completely opaque but it looks like the output you've got (in the image you posted) is already telling you where the problems are.

bigblondewolf · 2013-09-19 22:06:08

You are right - the halt is caused by the error message on the screenshot. I had the feeling that plymouth was not to blame, but /srv/nfs/trunk got mounted eventually and that mislead me.

/srv/nfs/trunk is actually a --bind, so I suppose systemd is trying to mount it asynchronously before the actual filesystem, hence it's failed and dead:

[1] root@xxxlinux : /home/ognyan $ systemctl status srv-nfs-trunk.mount
srv-nfs-trunk.mount - /srv/nfs/trunk
   Loaded: loaded (/etc/fstab)
   Active: inactive (dead) since Thu 2013-09-19 10:28:11 EEST; 14h ago
    Where: /srv/nfs/trunk
     What: /mnt/trunk
  Process: 413 ExecMount=/bin/mount /mnt/trunk /srv/nfs/trunk -t none -o bind (code=exited, status=0/SUCCESS)

Sep 19 10:28:11 xxxlinux systemd[1]: Mounted /srv/nfs/trunk.

(this is the output via ssh after the system boots completely, but it is similar on emergency mode)

After Ctrl-d'ing systemd makes a second attempt, and time successful.

I removed the bind mounts from fstab and now my system boots. Still two questions:

- binds are at the end of my fstab, how do I order mounts so these wait?
- why did it work before the update?

Many thanks for the help!

Last edited by bigblondewolf (2013-09-19 22:07:36)

cfr · 2013-09-19 22:12:50

Please post fstab.

Last edited by cfr (2013-09-19 22:13:12)

bigblondewolf · 2013-09-19 22:14:34

# Intel SSD 1
UUID=775642a3-f485-4cc6-b9d5-c336aa276c39       /                           xfs         rw,relatime,attr2,inode64,noquota       0 1

# Intel SSD 2
UUID=75232369-0f3f-45f5-99fe-234089a1ea8e       /boot                       xfs         rw,relatime,attr2,inode64,noquota       0 2

# Intel SSD 3
UUID=595fa411-2c31-48e1-960b-fad259460248       /home                       xfs         rw,relatime,attr2,inode64,noquota       0 2

# WD 1T 1
UUID=4fcb28a9-66fe-4463-9f31-e325d62d5017   /mnt/wd1                xfs         rw,relatime,attr2,inode64,noquota   0 2
# WD 1T 2
UUID=4803c5db-72a3-4cd5-ad94-f7de2de2ea09   /mnt/wd2                xfs         rw,relatime,attr2,inode64,noquota   0 2
# WD 1T 3
UUID=b937ce12-31a4-4816-9630-d53cab774a7e   /mnt/wd3                xfs         rw,relatime,attr2,inode64,noquota   0 2

# SG 320G 1 1
UUID=76fd28c6-1541-40bd-958b-cd05d787ff7b   /mnt/sg1_1              xfs         rw,relatime,attr2,inode64,noquota   0 2

# SG 320G 2 1
UUID=789845B4984571A4                       /mnt/sg2_5              ntfs-3g     uid=1000,gid=users,dmask=022,fmask=133  0 0
UUID=32BC47F9BC47B5DF                       /mnt/sg2_6              ntfs-3g     uid=1000,gid=users,dmask=022,fmask=133  0 0

tmpfs                                       /tmp                    tmpfs           nodev,nosuid,size=20G            0  0

# NFS mounts
/mnt/multimedia                             /srv/nfs/multimedia     none bind                                0 0
/mnt/images                                 /srv/nfs/images         none bind                                0 0
/mnt/trunk                                  /srv/nfs/trunk          none bind                                0 0

[2] ognyan@xxxlinux : ~ >ll /mnt/
total 40K
drwxr-xr-x   8 root   root   124 Jun 13 01:22 .
drwxr-xr-x  18 root   root  4.0K Sep 17 23:52 ..
lrwxrwxrwx   1 root   root     3 Jun  4 17:16 images -> wd1
lrwxrwxrwx   1 root   root     3 Jun  4 17:16 multimedia -> wd2
drwxr-xr-x  23 ognyan users 4.0K Sep 19 14:37 sg1_1
drwxr-xr-x   1 ognyan users 8.0K Mar 30 21:28 sg2_5
drwxr-xr-x   1 ognyan users  12K Mar 30 21:28 sg2_6
lrwxrwxrwx   1 root   root     3 Jun  3 21:27 srv -> wd3
lrwxrwxrwx   1 root   root     5 Jun  4 17:15 trunk -> sg1_1
drwxr-sr-x 203 ognyan users 8.0K Aug  4 20:53 wd1
drwxr-xr-x   5 root   root    41 Sep 19 10:32 wd2
drwxr-xr-x   8 root   root    76 Aug 31 23:13 wd3

Last edited by bigblondewolf (2013-09-19 22:15:36)

cfr · 2013-09-19 23:27:35

Can you use the network options along with bind so that it won't hang if the network isn't available, for example? (https://wiki.archlinux.org/index.php/NF … b_settings)

bigblondewolf · 2013-09-20 05:34:35

This is actually the nfs server and /srv/nfs is the root of the exports. It should not depend on network or I am wrong?

bigblondewolf · 2013-10-08 21:41:02

Hi guys, my problem still persists, even after latest updates. As I mentioned, this is the *nfs server*, so this is a local mount. What may be the reason to fail booting when mounting --bind a local directory?
btw since it is definitely not a plymouth question, should I change the topic?

cfr · 2013-10-08 21:59:46

Just curious, why the symbolic links under /mnt? Why not just bind the mountpoints directly?

By the way, you don't probably want the tmpfs line in fstab.

bigblondewolf · 2013-10-09 20:56:59

cfr wrote:

By the way, you don't probably want the tmpfs line in fstab.

Default tmpfs on /tmp was too small for me once and I extended it to something that would be enough, maybe I'll remove it until I need more than 4G again.

cfr wrote:

Just curious, why the symbolic links under /mnt? Why not just bind the mountpoints directly?

I was thinking to replace the symlinks with binds, but kept them so logical mount names stand out. As a matter of fact today I switched them and the system started smoothly (yay!). I think this topic can go to its [SOLVED] state. I'm still curious what broke this setup after the update though, but it doesn't matter much, lesson is learned.

Thanks a lot for your help!

Arch Linux

#1 2013-09-17 22:24:47

Failed at step EXEC spawning /bin/plymouth: No such file or directory

#2 2013-09-19 01:03:37

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#3 2013-09-19 07:37:42

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#4 2013-09-19 21:40:41

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#5 2013-09-19 22:06:08

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#6 2013-09-19 22:12:50

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#7 2013-09-19 22:14:34

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#8 2013-09-19 23:27:35

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#9 2013-09-20 05:34:35

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#10 2013-10-08 21:41:02

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#11 2013-10-08 21:59:46

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

#12 2013-10-09 20:56:59

Re: Failed at step EXEC spawning /bin/plymouth: No such file or directory

Board footer