[SOLVED]Create Volatile Files& Directories& Login Service fail at boot

Slashee the Cow · 2015-04-12 15:32:20

So I just installed Arch and got past the bootloader (which itself was an effort). Now that grub actually works, the first time it got into Arch, I got the bit where you set the locale and timezone and root password and stuff, and then after that most things seemed to start except I got:

[FAILED] Failed to start Create Volatile Files and Directories
See "systemctl status systemd-tmpfiles-setup.service" for details.
<snip some stuff that loads fine>
[FAILED] Failed to start Login Service.
See "systemctl status systemd-logind.service" for details.

and then a blinking white cursor. I left it for half an hour to see if it'd do anything; it didn't.

It doesn't seem to respond to any input (tried enter, esc, various letters, various mostly random ctrl+key combinations, ctrl+alt+f-keys), the only thing it responds to is ctrl+alt+del, to which it reboots properly (and doesn't come up with the first run thing again). In case it matters, my setup is (mostly) what's explained in my other thread (not sure if it matters): I have a btrfs partition with four subvolumes, for /, /etc, /home and /var; and an ext4 partition for /boot. I'd be happy to provide logs or whatever (whatever I can get from booting a live instance of Arch and mounting my installation) but I'd have no idea where to look (that is to say: while I'm not especially new at Linux, all the distros I've run before are much more user friendly, so I don't need to worry about this stuff).

Solved: Pretty sure this was due to the btrfs setup I had. Specifically, I had /etc as its own subvolume, which was a child of the main volume. I knew I was on to something when I didn't give /etc its own subvolume, and it worked. I speculate about the cause below, but basically it seems to work when I made the subvolume for /etc a child of the subvolume for /, so that when the subvolume for / gets mounted, the subvolume for /etc gets automatically mounted at the same time.

Last edited by Slashee the Cow (2015-04-14 13:46:07)

Head_on_a_Stick · 2015-04-12 15:59:11

Slashee the Cow wrote:

Now that grub actually works, the first time it got into Arch, I got the bit where you set the locale and timezone and root password and stuff

Which guide are you following?

Locales & timezones are generally set up before GRUB (or any other boot loader/managers) is installed & configured.

Slashee the Cow · 2015-04-12 16:28:51

The beginners guide on the wiki. I did set up the locale and time zone during installation (while I was chrooted into my install) but when I first booted it (well, successfully; see: grub troubles) there was something that ran asking me to pick my locale (I only had a choice of the two I'd uncommented during setup, en_US.UTF-8 and en_AU.UTF-8), my time zone (from a rather large numerical list) and to pick a root password. Then the things I mentioned above failed to work, and on subsequent boots they still fail but I didn't get the first run thing again.

Head_on_a_Stick · 2015-04-12 16:50:05

Slashee the Cow wrote:

when I first booted it [...] there was something that ran asking me to pick my locale (I only had a choice of the two I'd uncommented during setup, en_US.UTF-8 and en_AU.UTF-8), my time zone (from a rather large numerical list) and to pick a root password.

I've installed Arch quite a few times now and that has never happened to me...

Load up the Arch live ISO, mount all of your partitions, use `arch-chroot` and then please post the output of:

uname -a
lsblk -f
cat /etc/fstab

Slashee the Cow · 2015-04-13 00:32:18

Head_on_a_Stick wrote:

I've installed Arch quite a few times now and that has never happened to me...

Maybe it's new or something? I don't know.

uname -a

Linux archiso 3.19.2-1-ARCH #1 SMP PREEMPT Wed Mar 18 16:21:02 CET 2015 x86_64 GNU/Linux

lsblk -f

NAME            FSTYPE LABEL UUID MOUNTPOINT
sda                               
├─sda1                            /boot
├─sda2                            
├─sda3                            
├─sda4                            [SWAP]
├─sda5                            /home
└─sda6                            
sdb                               
└─sdb1                            
sdc                               
└─sdc1                            
sdd                               
└─sdd1                            
sde                               
├─sde1                            
├─sde2                            
├─sde3                            
└─sde5                            
sdf                               
└─sdf1                            
sdg                               
└─sdg1                            
sr0                               
loop0                             
loop1                             
└─arch_airootfs                   /etc/resolv.conf
loop2                             
└─arch_airootfs                   /etc/resolv.conf

(I don't know why it's not actually displaying the fstypes or anything, although it does when it's not chrooted)

cat /etc/fstab

# /dev/sda5 LABEL=Linux\134x20btrfs
UUID=b21e6211-5e7a-4769-9f53-b9b5d554a242	/         	btrfs     	rw,relatime,space_cache	0 0

# /dev/sda1 LABEL=Linux\134x20/boot
UUID=c6fcd25d-72ed-4e36-b344-40cf3762f1af	/boot     	ext4      	rw,relatime,stripe=4,data=ordered	0 2

# /dev/sda5 LABEL=Linux\134x20btrfs
UUID=b21e6211-5e7a-4769-9f53-b9b5d554a242	/etc      	btrfs     	rw,relatime,space_cache,subvol=etc	0 0

# /dev/sda5 LABEL=Linux\134x20btrfs
UUID=b21e6211-5e7a-4769-9f53-b9b5d554a242	/var      	btrfs     	rw,relatime,space_cache,subvol=var	0 0

# /dev/sda5 LABEL=Linux\134x20btrfs
UUID=b21e6211-5e7a-4769-9f53-b9b5d554a242	/home     	btrfs     	rw,relatime,space_cache,subvol=home	0 0

# /dev/sda4 LABEL=Linux\134x20swap
UUID=c7b1f7e1-8a63-43ff-892e-74013a5d6c12	none      	swap      	defaults  	0 0

EscapedNull · 2015-04-13 04:36:26

Try booting into recovery media and running

journalctl -b -D /mnt/var/log/journal -p 3 -x

This should give you more detailed error messages about why the system failed to boot. See journalctl(1).

Slashee the Cow · 2015-04-13 05:12:43

journalctl -b -D /mnt/var/log/journal -p 3 -x only returns

No journal files were found.

Wasn't sure if it'd make a difference, but got the same when I chrooted in to the install and tried that (without the /mnt obviously).

I'm honestly not sure what I expected to find but I looked in /var/log/journal of my installation and it only seemed to contain an empty folder called remote. Not sure if that's a good sign or (probably more likely) a very bad sign.

Also worth noting: I mentioned in my other thread that I recreated what I did in a virtual machine to see if grub not working was something I did or a problem unique to the real setup (for what it's worth, grub worked fine in the virtual machine). However the virtual machine fails to boot in the same way as the real install so in this case I'm guessing it's something I did.

Head_on_a_Stick · 2015-04-13 11:53:31

Slashee the Cow wrote:

lsblk -f

NAME            FSTYPE LABEL UUID MOUNTPOINT
sda                               
├─sda1                            /boot
├─sda2                            
├─sda3                            
├─sda4                            [SWAP]
├─sda5                            /home
└─sda6

You do not appear to have your root, /etc or /var subvolumes mounted.

I have very little experience of BTRFS so I don't know if that is normal.

What is the output of:

# btrfs subvolume list -p .

Slashee the Cow · 2015-04-13 12:22:39

It's normal (at least, I'm pretty sure it is). I've run lsblk many times now and it always only seems to show one mount point if you have multiple subvolumes mounted. Just in case you can see something I don't though, here's the subvolume list:
btrfs subvolume list -p .

ID 257 gen 139 parent 5 top level 5 path root
ID 258 gen 7 parent 5 top level 5 path home
ID 259 gen 139 parent 5 top level 5 path var
ID 260 gen 127 parent 5 top level 5 path etc

Not sure if it helps, but after I mount everything and chroot into the installation mount outputs

/dev/sda5 on / type btrfs (rw,relatime,space_cache)
/dev/sda1 on /boot type ext4 (rw,relatime,stripe=4,data=ordered)
/dev/sda5 on /etc type btrfs (rw,relatime,space_cache)
/dev/sda5 on /home type btrfs (rw,relatime,space_cache)
/dev/sda5 on /var type btrfs (rw,relatime,space_cache)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4069592k,nr_inodes=1017398,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
tmp on /tmp type tmpfs (rw,nosuid,nodev)
/dev/mapper/arch_airootfs on /etc/resolv.conf type ext4 (rw,relatime)

dkulinski · 2015-04-13 15:56:18

I'm having this same issue. I have installed on both bare metal and a VM and the same error pops up. I am using BTRFS with subvolumes.

When I boot from the ISO there are no Journal files created so I can't troubleshoot the boot. When I boot into single user mode, on the bare metal machine I get an error about not being able to open the password db and on the VM when I log in I get the error: sulogin: crypt failed: Invalid argument.

EscapedNull · 2015-04-13 19:08:29

I'm shooting in the dark here, but can you boot from rescue media and post the output of some of these commands?

/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev --root=/mnt

arch-chroot /mnt
/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev

/usr/lib/systemd/system-generators/systemd-fstab-generator
head -n 50 /tmp/*.mount

With the little amount of information we have to go on right now, I can't make any more intelligent guesses. Hopefully one of these will point us in the right direction.

Edit: use `echo $?` to check the exit code of each command. Some of them might not output errors.

Last edited by EscapedNull (2015-04-13 19:10:38)

dkulinski · 2015-04-13 22:07:21

This is run from my VM as I don't have access to the bare metal at the moment.

The first command runs without any output and has a 0 return code.

The second command fails:

root@archiso ~ # arch-chroot /mnt
sh-4.3# /usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev
Failed to create file /sys/devices/system/cpu/microcode/reload: Read-only file system

==> /tmp/boot.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/vda1
Where=/boot
Type=btrfs
Options=rw,relatime,space_cache,subvol=boot

==> /tmp/etc.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/vda1
Where=/etc
Type=btrfs
Options=rw,relatime,space_cache,subvol=etc

==> /tmp/home.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/vda1
Where=/home
Type=btrfs
Options=rw,relatime,space_cache,subvol=home

==> /tmp/-.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/vda1
Where=/
Type=btrfs
Options=rw,relatime,space_cache

==> /tmp/var.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/vda1
Where=/var
Type=btrfs
Options=rw,relatime,space_cache,subvol=var

jedijimi · 2015-04-14 00:17:29

run 'sudo pwck' and 'grpck'

Last edited by jedijimi (2015-04-14 00:17:54)

Slashee the Cow · 2015-04-14 00:40:02

dkulinski wrote:

This is run from my VM as I don't have access to the bare metal at the moment.
The first command runs without any output and has a 0 return code.
The second command fails:
root@archiso ~ # arch-chroot /mnt
sh-4.3# /usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev
Failed to create file /sys/devices/system/cpu/microcode/reload: Read-only file system

Same here (and I ran those on the bare metal... if I'm using that term correctly). My results for the third one are only slightly different (I'm using UUIDs and my /boot is ext4, not btrfs):

==> /tmp/boot.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
RequiresOverridable=systemd-fsck@dev-disk-by\x2duuid-c6fcd25d\x2d72ed\x2d4e36\x2db344\x2d40cf3762f1af.service
After=systemd-fsck@dev-disk-by\x2duuid-c6fcd25d\x2d72ed\x2d4e36\x2db344\x2d40cf3762f1af.service

[Mount]
What=/dev/disk/by-uuid/c6fcd25d-72ed-4e36-b344-40cf3762f1af
Where=/boot
Type=ext4
Options=rw,relatime,stripe=4,data=ordered

==> /tmp/etc.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/disk/by-uuid/b21e6211-5e7a-4769-9f53-b9b5d554a242
Where=/etc
Type=btrfs
Options=rw,relatime,space_cache,subvol=etc

==> /tmp/home.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/disk/by-uuid/b21e6211-5e7a-4769-9f53-b9b5d554a242
Where=/home
Type=btrfs
Options=rw,relatime,space_cache,subvol=home

==> /tmp/-.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/disk/by-uuid/b21e6211-5e7a-4769-9f53-b9b5d554a242
Where=/
Type=btrfs
Options=rw,relatime,space_cache

==> /tmp/var.mount <==
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
What=/dev/disk/by-uuid/b21e6211-5e7a-4769-9f53-b9b5d554a242
Where=/var
Type=btrfs
Options=rw,relatime,space_cache,subvol=var

jedijimi wrote:

run 'sudo pwck' and 'grpck'

Those both give no output and when I check with 'echo $?' they only returned 0.

Slashee the Cow · 2015-04-14 02:13:48

So I think I might have found the problem. I was playing around installing Fedora in a VM and its installer won't let you have /etc as a separate mountpoint to / (not even as a btrfs subvolume in the same volume as /) - when I set up /etc as a separate subvolume in Arch I was just following the wiki. So I created a separate Arch VM, still installed on btrfs, but with subvolumes only for /, /home and /var (i.e. didn't create a separate subvolume for /etc) and /boot still its own ext4 partition. Installed like normal (something I've gotten quick at, considering I've tried a few times now), rebooted the VM, and... bam, straight to a login prompt (didn't even get the locale/timezone thing that Head_on_a_Stick has never seen, which would explain never having seen it).

So I guess that might be how to stop it happening. However, I wouldn't mind having /etc on a separate subvolume (can easily make backups of it just by making a snapshot)... any idea how I can make this happen?

Last edited by Slashee the Cow (2015-04-14 02:15:44)

Slashee the Cow · 2015-04-14 13:41:31

So, I think I might have solved my problem (don't know about you dkulinski, but I sure hope this helps you too). If you'll allow me to completetely speculate about the cause based on my limited knowledge of Linux, then my guess would be that although grub knows how to mount /, it's not mounting the other subvolumes because /etc isn't mounted (so it can't read /etc/fstab) and it can't mount /etc because the instructions for how to do so are locked away in /etc/fstab. Could be entirely wrong there, just my guess.

Anyway, I fixed it by, instead of creating all four subvolumes (root_dir, etc, home and var) as subvolumes at the root level, I made the etc subvolume a child of root_dir - for anyone who's figuring out stuff about btrfs at the same time I am, that means instead of doing:

btrfs subvolume create root_dir
btrfs subvolume create etc
btrfs subvolume create home
btrfs subvolume create var

to create all my subvolumes, I did

btrfs subvolume create root_dir
btrfs subvolume create root_dir/etc
btrfs subvolume create home
btrfs subvolume create var

... although I suppose if you've already gone to the effort of installing it you could move the subvolume - which I think you do just by mounting the main btrfs volume (use subvolid="0" if you set a default subvolume like I did) and then just use mv to move the subvolume, and remove its entry from /etc/fstab.

Since child subvolumes automatically get mounted when their parents do, that means root_dir/etc gets mounted at the same time root_dir does, and that seems to work (I've only tested this so far in a VM I created just to test it, so it's not like I had a complicated install to test, but it gets to the login prompt, which is far more than it did when it wasn't working).

Arch Linux

#1 2015-04-12 15:32:20

[SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#2 2015-04-12 15:59:11

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#3 2015-04-12 16:28:51

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#4 2015-04-12 16:50:05

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#5 2015-04-13 00:32:18

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#6 2015-04-13 04:36:26

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#7 2015-04-13 05:12:43

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#8 2015-04-13 11:53:31

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#9 2015-04-13 12:22:39

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#10 2015-04-13 15:56:18

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#11 2015-04-13 19:08:29

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#12 2015-04-13 22:07:21

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#13 2015-04-14 00:17:29

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#14 2015-04-14 00:40:02

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#15 2015-04-14 02:13:48

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

#16 2015-04-14 13:41:31

Re: [SOLVED]Create Volatile Files& Directories& Login Service fail at boot

Board footer