You are not logged in.
Hi guys, Yet Another Plymouth Problem:
I just did a system update. Haven't done this for several months and problems with /bin happened, as expected. Anyway, after update completed, the system won't boot normally:
If I type ^d boot continues with no problems and all services are started as expected. I have trouble finding what causes the system to go to emergency mode. The /srv/nfs/trunk/ seems to be mounted at a second attempt.
An excerpt from journalctl -xb, right after the xfs messages :
Sep 18 00:55:50 xxxlinux systemd[1]: Startup finished in 2.322s (kernel) + 1.350s (userspace) = 3.672s.
-- Subject: System start-up is now complete
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- All system services necessary queued for starting at boot have been
-- successfully started. Note that this does not mean that the machine is
-- now idle as services might still be busy with completing start-up.
--
-- Kernel start-up required 2322291 microseconds.
--
-- Initial RAM disk start-up required INITRD_USEC microseconds.
--
-- Userspace start-up required 1350682 microseconds.
Sep 18 00:55:50 xxxlinux systemd[168]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
-- Subject: Process /bin/plymouth could not be executed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/641257651c1b4ec9a8624d7a40a9e1e7
--
-- The process /bin/plymouth could not be executed and failed.
--
-- The error number returned while executing this process is 2.
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Version 2013.1.13 external FUSE 29
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Mounted /dev/sdb6 (Read-Write, label "", NTFS 3.1)
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Cmdline options: rw,uid=1000,gid=100,dmask=022,fmask=133
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Mount options: rw,allow_other,nonempty,relatime,default_permissions,fsname=/dev/sdb6,blkdev,blk
Sep 18 00:55:50 xxxlinux ntfs-3g[294]: Global ownership and permissions enforced, configuration type 7
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:0: Direct-Access Generic USB SD Reader 1.00 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:1: Direct-Access Generic USB CF Reader 1.01 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:0: [sde] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:2: Direct-Access Generic USB SM Reader 1.02 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:1: [sdf] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: scsi 6:0:0:3: Direct-Access Generic USB MS Reader 1.03 PQ: 0 ANSI: 0
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:2: [sdg] Attached SCSI removable disk
Sep 18 00:55:50 xxxlinux kernel: sd 6:0:0:3: [sdh] Attached SCSI removable disk
After I press ^d:
Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Sound Card.
-- Subject: Unit sound.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sound.target has begun shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopped target Sound Card.
-- Subject: Unit sound.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/9d1aaa27d60140bd96365438aad20286
--
-- Unit sound.target has finished shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Emergency Mode.
-- Subject: Unit emergency.target has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit emergency.target has begun shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopped target Emergency Mode.
-- Subject: Unit emergency.target has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/9d1aaa27d60140bd96365438aad20286
--
-- Unit emergency.target has finished shutting down.
Sep 18 00:56:21 xxxlinux systemd[1]: Stopping Emergency Shell...
-- Subject: Unit emergency.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit emergency.service has begun shutting down.
And the system boots.
I found a thread explaining that there is a sysV runlevel specified in kernel boot line. I am not sure if I am looking at the proper file, but my /boot/grub/grub.cfg does not contain "runlevel".
I also found it may be a problem of software RAID, but I am not using it neither LVM, plain mounts by UUID.
I ran pacman -S linux, /etc/mkinitcpio.conf does not contain a reference to plymouth: HOOKS="base udev autodetect modconf block filesystems keyboard fsck"
Just checked with my laptop, both computers run systemd-204-3, neither have /bin/plymouth, but the laptop does not fail with this message
Just tell me if you need more information.
Offline
What is your kernel command line? If it ends with a number such as "3" or "5", that's a runlevel and you need to change it or get rid of it. If it is "5", you can just delete it - systemd will default to that anyway. If it is "3" replace it with systemd.unit=multi-user.target.
EDIT: I thought this change came in systemd 207. At any rate, if you are not using 207, you should update.
Last edited by cfr (2013-09-19 01:04:48)
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
What is your kernel command line? If it ends with a number such as "3" or "5", that's a runlevel and you need to change it or get rid of it. If it is "5", you can just delete it - systemd will default to that anyway. If it is "3" replace it with systemd.unit=multi-user.target.
Should this be in grub.cfg? The only thing that looks like a kernel command line there is within the menuentry block:
linux /vmlinuz-linux root=UUID=775642a3-f485-4cc6-b9d5-c336aa276c39 rw
EDIT: I thought this change came in systemd 207. At any rate, if you are not using 207, you should update.
I thought I did a full system update (pacman -Syu) on Monday, systemd was still 204. I did an update again, now at 207 the problem persists.
Offline
Did you execute the command systemd suggests to get information about why srv-nfs-trunk.mount is failing?
The reason I asked about runlevels was because the plymouth error can indicate that problem and is completely opaque but it looks like the output you've got (in the image you posted) is already telling you where the problems are.
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
You are right - the halt is caused by the error message on the screenshot. I had the feeling that plymouth was not to blame, but /srv/nfs/trunk got mounted eventually and that mislead me.
/srv/nfs/trunk is actually a --bind, so I suppose systemd is trying to mount it asynchronously before the actual filesystem, hence it's failed and dead:
[1] root@xxxlinux : /home/ognyan $ systemctl status srv-nfs-trunk.mount
srv-nfs-trunk.mount - /srv/nfs/trunk
Loaded: loaded (/etc/fstab)
Active: inactive (dead) since Thu 2013-09-19 10:28:11 EEST; 14h ago
Where: /srv/nfs/trunk
What: /mnt/trunk
Process: 413 ExecMount=/bin/mount /mnt/trunk /srv/nfs/trunk -t none -o bind (code=exited, status=0/SUCCESS)
Sep 19 10:28:11 xxxlinux systemd[1]: Mounted /srv/nfs/trunk.
(this is the output via ssh after the system boots completely, but it is similar on emergency mode)
After Ctrl-d'ing systemd makes a second attempt, and time successful.
I removed the bind mounts from fstab and now my system boots. Still two questions:
- binds are at the end of my fstab, how do I order mounts so these wait?
- why did it work before the update?
Many thanks for the help!
Last edited by bigblondewolf (2013-09-19 22:07:36)
Offline
Please post fstab.
Last edited by cfr (2013-09-19 22:13:12)
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
# Intel SSD 1
UUID=775642a3-f485-4cc6-b9d5-c336aa276c39 / xfs rw,relatime,attr2,inode64,noquota 0 1
# Intel SSD 2
UUID=75232369-0f3f-45f5-99fe-234089a1ea8e /boot xfs rw,relatime,attr2,inode64,noquota 0 2
# Intel SSD 3
UUID=595fa411-2c31-48e1-960b-fad259460248 /home xfs rw,relatime,attr2,inode64,noquota 0 2
# WD 1T 1
UUID=4fcb28a9-66fe-4463-9f31-e325d62d5017 /mnt/wd1 xfs rw,relatime,attr2,inode64,noquota 0 2
# WD 1T 2
UUID=4803c5db-72a3-4cd5-ad94-f7de2de2ea09 /mnt/wd2 xfs rw,relatime,attr2,inode64,noquota 0 2
# WD 1T 3
UUID=b937ce12-31a4-4816-9630-d53cab774a7e /mnt/wd3 xfs rw,relatime,attr2,inode64,noquota 0 2
# SG 320G 1 1
UUID=76fd28c6-1541-40bd-958b-cd05d787ff7b /mnt/sg1_1 xfs rw,relatime,attr2,inode64,noquota 0 2
# SG 320G 2 1
UUID=789845B4984571A4 /mnt/sg2_5 ntfs-3g uid=1000,gid=users,dmask=022,fmask=133 0 0
UUID=32BC47F9BC47B5DF /mnt/sg2_6 ntfs-3g uid=1000,gid=users,dmask=022,fmask=133 0 0
tmpfs /tmp tmpfs nodev,nosuid,size=20G 0 0
# NFS mounts
/mnt/multimedia /srv/nfs/multimedia none bind 0 0
/mnt/images /srv/nfs/images none bind 0 0
/mnt/trunk /srv/nfs/trunk none bind 0 0
[2] ognyan@xxxlinux : ~ >ll /mnt/
total 40K
drwxr-xr-x 8 root root 124 Jun 13 01:22 .
drwxr-xr-x 18 root root 4.0K Sep 17 23:52 ..
lrwxrwxrwx 1 root root 3 Jun 4 17:16 images -> wd1
lrwxrwxrwx 1 root root 3 Jun 4 17:16 multimedia -> wd2
drwxr-xr-x 23 ognyan users 4.0K Sep 19 14:37 sg1_1
drwxr-xr-x 1 ognyan users 8.0K Mar 30 21:28 sg2_5
drwxr-xr-x 1 ognyan users 12K Mar 30 21:28 sg2_6
lrwxrwxrwx 1 root root 3 Jun 3 21:27 srv -> wd3
lrwxrwxrwx 1 root root 5 Jun 4 17:15 trunk -> sg1_1
drwxr-sr-x 203 ognyan users 8.0K Aug 4 20:53 wd1
drwxr-xr-x 5 root root 41 Sep 19 10:32 wd2
drwxr-xr-x 8 root root 76 Aug 31 23:13 wd3
Last edited by bigblondewolf (2013-09-19 22:15:36)
Offline
Can you use the network options along with bind so that it won't hang if the network isn't available, for example? (https://wiki.archlinux.org/index.php/NF … b_settings)
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
This is actually the nfs server and /srv/nfs is the root of the exports. It should not depend on network or I am wrong?
Offline
Hi guys, my problem still persists, even after latest updates. As I mentioned, this is the *nfs server*, so this is a local mount. What may be the reason to fail booting when mounting --bind a local directory?
btw since it is definitely not a plymouth question, should I change the topic?
Offline
Just curious, why the symbolic links under /mnt? Why not just bind the mountpoints directly?
By the way, you don't probably want the tmpfs line in fstab.
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
By the way, you don't probably want the tmpfs line in fstab.
Default tmpfs on /tmp was too small for me once and I extended it to something that would be enough, maybe I'll remove it until I need more than 4G again.
Just curious, why the symbolic links under /mnt? Why not just bind the mountpoints directly?
I was thinking to replace the symlinks with binds, but kept them so logical mount names stand out. As a matter of fact today I switched them and the system started smoothly (yay!). I think this topic can go to its [SOLVED] state. I'm still curious what broke this setup after the update though, but it doesn't matter much, lesson is learned.
Thanks a lot for your help!
Offline