Root Partition Warning on Boot

jgwaro · 2013-08-03 14:13:14

I have a fresh Arch installation. Am sure i followed the installation guide step by step but am getting this warning on every boot.

The root device is not configured to be mounted read-write! It may be fsck'd again later.

What does this mean? Any danger on my system?

Trilby · 2013-08-03 14:16:39

No danger - this is well covered in several threads recently. Please search before posting.

The freebie answer is you need to add "rw" to the kernel line of your boot loader. If you want more details or want to know why - find the other threads on it.

cfr · 2013-08-03 23:39:48

Also, if you are really a newbie you should not be using testing. If you are using testing, you should take the recommended precautions. Or at least look in the testing forum before posting.

If I am mistaken about this, please correct me, but I believe this change comes in systemd 206 and I still have 204 from the stable repos, having updated everything within the last couple of hours. So I can only assume the OP is using testing.

falconindy · 2013-08-03 23:53:24

cfr wrote:

If I am mistaken about this, please correct me, but I believe this change comes in systemd 206 and I still have 204 from the stable repos, having updated everything within the last couple of hours. So I can only assume the OP is using testing.

mkinitcpio 0.15 generates this warning.

cfr · 2013-08-04 00:10:57

Ah, I didn't know that. Thanks a lot. Guess I should change mine in that case, too.

That is, I knew the warning was from the initramfs but I thought it was to do with a change coming in systemd 206. Is the warning in this version of mkinitcpio anticipating that change?

Last edited by cfr (2013-08-04 00:26:09)

falconindy · 2013-08-04 00:33:08

cfr wrote:

Is the warning in this version of mkinitcpio anticipating that change?

Regardless of what precipitated the change (yes, it's systemd 206), mounting your disk read-only after fsck'ing it is odd choice.

cfr · 2013-08-04 00:36:47

I think I assumed that it had not yet been fsck'd at the relevant point in the boot sequence. I don't think I had any reason for thinking that except that everyone seemed to have 'ro' in the kernel command line. So I guess I just figured it was normal for Linux.

mcloaked · 2013-08-04 15:05:17

falconindy wrote:

cfr wrote:
Is the warning in this version of mkinitcpio anticipating that change?
Regardless of what precipitated the change (yes, it's systemd 206), mounting your disk read-only after fsck'ing it is odd choice.

OK so am I then correct if I interpret this as follows:

The current version of mkinitcpio (0.15) which is in [core] gives the warning about "The root device is not configured to be mounted read-write! It may be fsck'd again later."

So it is "suggesting" that the root device should be mounted rw but it is not clear that this means it is mounted ro "after" fsck'ing it (via systemd). There is more than one way to achieve this but one way would be to set the line with the "ro" in the section:

  message="$(gettext_printf "Loading Linux %s ..." "${version}")"
  sed "s/^/$submenu_indentation/" << EOF
        echo    '$(echo "$message" | grub_quote)'
        linux   ${rel_dirname}/${basename} root=${linux_root_device_thisversion} ro ${args}
EOF

in the file /etc/grub.d/10_linux from "ro" to "rw" - and if mkinitcpio is then executed then the root file system will be remounted rw after fsck'ing (by systemd) if there is no fsck mkinitcpio hook? Or at this point is the fsck hook still needed - that point is not clear to me.

I still have this file with "ro" but removed fsck HOOK from the mkinitcpio.conf file - and my systemd journal appears to show that fsck has indeed been done, and also that the mount command shows that the root system is indeed mounted just fine as rw!

 Aug 04 14:54:44 lapmike3 systemd-fsck[105]: /dev/sda8: clean, 381851/1026000 files, 3476621/40960
Aug 04 14:54:44 lapmike3 systemd-udevd[122]: specified group 'plugdev' unknown
Aug 04 14:54:44 lapmike3 systemd[1]: Started File System Check on Root Device.
Aug 04 14:54:44 lapmike3 systemd[1]: Starting Remount Root and Kernel File Systems...
Aug 04 14:54:44 lapmike3 systemd[1]: Started Setup Virtual Console.
Aug 04 14:54:45 lapmike3 systemd[1]: Started Remount Root and Kernel File Systems.
Aug 04 14:54:45 lapmike3 systemd[1]: Starting Local File Systems (Pre).
Aug 04 14:54:45 lapmike3 systemd[1]: Reached target Local File Systems (Pre).
Aug 04 14:54:45 lapmike3 systemd[1]: Mounting /tmp...

and

$ sudo systemctl status systemd-fsck-root
systemd-fsck-root.service - File System Check on Root Device
   Loaded: loaded (/usr/lib/systemd/system/systemd-fsck-root.service; static)
   Active: inactive (dead) since Sun 2013-08-04 14:54:44 BST; 3min 19s ago
     Docs: man:systemd-fsck@.service(8)
  Process: 105 ExecStart=/usr/lib/systemd/systemd-fsck (code=exited, status=0/SUCCESS)

Aug 04 14:54:44 lapmike3 systemd-fsck[105]: /dev/sda8: clean, 381851/1026000 files, 3476621...ts)
Aug 04 14:54:44 lapmike3 systemd[1]: Started File System Check on Root Device.

and the mount command shows:

/dev/sda8 on / type ext4 (rw,relatime,commit=600,data=ordered)

This setup does not give the warning about the file system not being mounted rewrite.

So is this still the wrong approach despite "ro" in the grub config file, and no fsck as a mkinitcpio hook? I guess the sequence of events and knowing exactly where in the boot sequence between the console listing of the systemd-fsck lines with each partition being "clean" and the running system showing that the root file system has indeed been remounted rw.

To me it is not clear (perhaps simply ignorance!) whether systemd now mounts the root file system as ro to do its systemd-fsck and then remounts rw and there are no further remounts or whether there are other mounts as ro for some people with an additional fsck initiated by grub with the root file system being rmounted ro for a second fsck, and then a third remount as rw for the running system? If there is a description of the boot process that makes the mount and file system check sequence clear for different scenarios that would be helpful as I can't interpret the available information to understand this - maybe it is just me who has less understanding the most of the others in this thread!

If someone who really understands this could post a clear answer it would help enormously - even though I know that the arch philosophy is that users read and learn and try to sort this kind of thing out for themselves. I tried but could not find a clear answer.

By the way my "kernel" line is default and I have not changed it at all.

Last edited by mcloaked (2013-08-04 15:07:47)

falconindy · 2013-08-04 15:26:58

I don't understand why this is so complicated. There's two players involved, and it's always been this way regardless of the versions involved (at at least as long as the fsck hook has existed):

1) mkinitcpio offers you to option to fsck your root device before mounting it via the fsck hook. If you do this, you should mount root read-write via the appropriate 'rw' option on the kernel commandline. Historically, you mounted root read-only so that it could be fsck'd later on in the bootstrap. If root has already been fsck'd, why would you mount it read-only just to be remounted read-write later on?

2) systemd will fsck all filesystems assuming three things are true: the filesystem has a fsck pass number greater than 0 (either from /etc/fstab or a user-supplied unit file), the filesystem is not already mounted read-write, and you've not elected to disable fsck entirely via the kernel commandline option fsck.mode=skip.

In short, these are the only 2 logical setups:
1) Use the 'fsck' hook, use 'rw' on the kernel commandline.
2) Don't use the 'fsck' hook, use 'ro' on the kernel commandline.

Both of these options result in your root device being checked exactly once. The unlisted option 3 which most people seem to be running into isn't harmful (which is why it's simply a warning). Again, mkinitcpio is pointing out that root might be fsck'd again. mkinitcpio has no way of knowing for sure what will happen after it calls switch_root and it's only rightfully warning you that your config is strange.

Editorial snark: If your bootloader makes this change difficult for you, maybe it's time to consider a different bootloader.

mcloaked · 2013-08-04 15:40:36

Thank you falconindy for the really spot-on reply that does indeed make a clear statement about what works and why.

Just an additional comment to your option 2) of your two logical setups - Don't use the fsck mkinitcpio hook, and leave the "default" ro option (i.e. what you get if you don't change the kernel line in the boot setup which is the default) (which is what I have), rather than specifically set it as "ro" is just as good. So with a default setup if fsck is one of the mkinitcpio HOOKS then simply take that out, and re-execute mkinitcpio.

I guess that your post will now allow this long running thread to be put to bed.

At least I can rest now knowing that by luck I did choose a setup that works in the one machine where I am using grub - though I will of course also have to check my uefi machines where grub is not even installed, and where I only use rEFInd to boot - but I believe I can sort this out now - and take the editorial "snark" into account!

x33a · 2013-08-04 15:40:39

@ falconindy,

I am using OpenRC, so, do I need to add rw to the kernel boot line?

If I am not mistaken, my root is already checked twice on boot (because I use the fsck hook).

cfr · 2013-08-04 16:15:47

I thought that earlier versions of systemd did not fsck the root volume if the initramfs had the fsck hook because it checked another flag somewhere which is now being deprecated. For example, my machine is booted with a kernel command line that includes ro (because I haven't rebooted since updating my config) and includes fsck in mkinitcpio.conf. However, grepping journalctl -b for systemd-fsck seems to indicate that systemd fsck'ed all volumes *except* root.

mcloaked · 2013-08-04 16:32:51

cfr wrote:

I thought that earlier versions of systemd did not fsck the root volume if the initramfs had the fsck hook because it checked another flag somewhere which is now being deprecated. For example, my machine is booted with a kernel command line that includes ro (because I haven't rebooted since updating my config) and includes fsck in mkinitcpio.conf. However, grepping journalctl -b for systemd-fsck seems to indicate that systemd fsck'ed all volumes *except* root.

I have a system with rEFInd as the boot loader - on this system it is running systemd 204 (current) - with the fsck hook still in mkinitcpio -

Grepping journalctl -b for fsck gives:

$ sudo journalctl -b|grep fsck
Aug 02 23:19:00 home1 systemd-fsck[196]: /dev/sda2: recovering journal
Aug 02 23:19:00 home1 systemd-fsck[208]: fsck.fat 3.0.22 (2013-07-19)
Aug 02 23:19:00 home1 systemd-fsck[208]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Aug 02 23:19:00 home1 systemd-fsck[208]: Automatically removing dirty bit.
Aug 02 23:19:00 home1 systemd-fsck[208]: Free cluster summary wrong (509943 vs. really 509945)
Aug 02 23:19:00 home1 systemd-fsck[208]: Auto-correcting.
Aug 02 23:19:00 home1 systemd-fsck[208]: Performing changes.
Aug 02 23:19:00 home1 systemd-fsck[208]: /dev/sdb1: 161 files, 1051/510996 clusters
Aug 02 23:19:00 home1 systemd-fsck[196]: /dev/sda2: clean, 136662/15122432 files, 9150323/60466432 blocks
Aug 04 10:29:38 home1 sudo[8035]: mike : TTY=pts/3 ; PWD=/home/mike ; USER=root ; COMMAND=/usr/bin/systemctl status systemd-fsck-root

the root partition is sdb2 which does not appear, possibly because with the fsck HOOK still in then either it has to be removed or the "ro" needs to be changed to "rw"!! So I guess as per falconindy's post having the "ro" default and leaving fsck in as a HOOK is a bad idea!

So this seems to be indicating you may well be correct! I may try to remove the fsck hook and check after re-running mkinitcpio and rebooting. I should also check the laptop where I have grub doing the boot which I referred to earlier in this thread.

Edited soon after the above post:

I did remove the fsck HOOK from mkinitcpio.conf on my uefi system - then ran mkinitcpio -p linux - and rebooted - now the systemd journal "does" include the filesystem check for the root partition:

$ sudo journalctl -b|grep fsck
Aug 04 17:35:37 home1 systemd-fsck[124]: /dev/sdb2: clean, 363857/1826816 files, 3425984/7303936 blocks
Aug 04 17:35:37 home1 systemd-fsck[205]: /dev/sda2: Superblock last write time is in the future.
Aug 04 17:35:37 home1 systemd-fsck[205]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.
Aug 04 17:35:37 home1 systemd-fsck[205]: /dev/sda2: clean, 136053/15122432 files, 9163820/60466432 blocks
Aug 04 17:35:37 home1 systemd-fsck[202]: fsck.fat 3.0.22 (2013-07-19)
Aug 04 17:35:37 home1 systemd-fsck[202]: Free cluster summary wrong (509943 vs. really 509945)
Aug 04 17:35:37 home1 systemd-fsck[202]: Auto-correcting.
Aug 04 17:35:37 home1 systemd-fsck[202]: Performing changes.
Aug 04 17:35:37 home1 systemd-fsck[202]: /dev/sdb1: 161 files, 1051/510996 clusters

So this again confirms that falconindy was correct and that cfr was right. So leaving the system as default was leaving my system without fsck running on the root partition!

Last edited by mcloaked (2013-08-04 16:41:54)

cfr · 2013-08-04 17:14:35

No. I believe that your root was being fsck'ed with the fsck initramfs hook but that fsck does not show in the journal and is not performed by systemd because it happens too early in the boot process i.e. before the partition is mounted.

That's why it is OK to have the fsck hook and rw on the kernel command line - your root partition will get fsck'ed once early in boot and systemd will not later fsck it.

Or you can take the fsck hook out and use ro - then your root will be fsck'ed once later on by systemd. (Assuming appropriate option in fstab and you haven't disabled systemd-fsck altogether etc.)

If you have the fsck hook and ro then in earlier versions of systemd, I *think* systemd would not fsck a second time because the initramfs fsck hook set a flag which told it not to. But in the newer versions of systemd (coming soon), it would be checked twice: once early by the initramfs hook and once later by systemd (because you used ro and that's all it will check together with fstab options etc.).

I could be wrong about this but that's how I understand it at the moment. That is, there are two points at which root can be fsck'ed: one is early using the initramfs fsck hook; the other is later by systemd. What you don't want to do is drop the hook and use rw (because root won't get fsck'ed at all) or use the hook and ro (because root will get fsck'ed twice though this is better than not at all).

mcloaked · 2013-08-04 17:26:15

cfr wrote:

No. I believe that your root was being fsck'ed with the fsck initramfs hook but that fsck does not show in the journal and is not performed by systemd because it happens too early in the boot process i.e. before the partition is mounted.
That's why it is OK to have the fsck hook and rw on the kernel command line - your root partition will get fsck'ed once early in boot and systemd will not later fsck it.
Or you can take the fsck hook out and use ro - then your root will be fsck'ed once later on by systemd. (Assuming appropriate option in fstab and you haven't disabled systemd-fsck altogether etc.)
If you have the fsck hook and ro then in earlier versions of systemd, I *think* systemd would not fsck a second time because the initramfs fsck hook set a flag which told it not to. But in the newer versions of systemd (coming soon), it would be checked twice: once early by the initramfs hook and once later by systemd (because you used ro and that's all it will check together with fstab options etc.).
I could be wrong about this but that's how I understand it at the moment. That is, there are two points at which root can be fsck'ed: one is early using the initramfs fsck hook; the other is later by systemd. What you don't want to do is drop the hook and use rw (because root won't get fsck'ed at all) or use the hook and ro (because root will get fsck'ed twice though this is better than not at all).

OK - though in a way the advantage of having systemd do the fsck is that it is at least in the journal as a logged record - whereas having an early initramfs fsck on the root partition means it would not have a log record (unless you allow the messages file to record it I guess) - I only have the systemd journal now.

So it is an individual choice as falconindy said originally, but it is good to clarify how the process works and how to choose the sensible options.

falconindy · 2013-08-04 17:30:55

mcloaked wrote:

OK - though in a way the advantage of having systemd do the fsck is that it is at least in the journal as a logged record - whereas having an early initramfs fsck on the root partition means it would not have a log record (unless you allow the messages file to record it I guess) - I only have the systemd journal now.

If you want logging in the initramfs, add "rd.log" to your command line. It'll be dumped into kmsg by default (which means it'll also end up in the journal). mkinticpio(8) details this behavior.

mcloaked · 2013-08-04 19:12:59

falconindy wrote:

mcloaked wrote:
OK - though in a way the advantage of having systemd do the fsck is that it is at least in the journal as a logged record - whereas having an early initramfs fsck on the root partition means it would not have a log record (unless you allow the messages file to record it I guess) - I only have the systemd journal now.
If you want logging in the initramfs, add "rd.log" to your command line. It'll be dumped into kmsg by default (which means it'll also end up in the journal). mkinticpio(8) details this behavior.

Thank you - that is useful additional control.

WonderWoofy · 2013-08-04 19:18:03

falconindy wrote:

If you want logging in the initramfs, add "rd.log" to your command line. It'll be dumped into kmsg by default (which means it'll also end up in the journal). mkinticpio(8) details this behavior.

I was trying to play with the rd.log parameter, and tried setting rd.log=file, which should put the log into /run/initramfs. But when I do this, it doesn't seem to work, and in fact I don't even have a /run/initramfs directory. Can anyone else get this to work? Am I just an idiot and doing something wrong? Or is this particular piece borked somehow?

cfr · 2013-08-04 19:47:35

I definitely have a /run/initramfs directory although I have not yet rebooted with the rd.log option. (I've added it to my kernel command lines but haven't yet tested it out and I've let it log to the default i.e. console for LTS and console/kmsg for current.)

 ls /run/initramfs/
VERSION  buildconfig  dev/  hooks/  init_functions  lib64@     proc/  root-fsck  sbin@      sys/  usr/
bin@     config       etc/  init*   lib@            new_root/  root/  run/       shutdown*  tmp/

EDIT: Try adding the shutdown hook to your initramfs. According to the info, that hook copies the contents of the initramfs to /run/initramfs for use on shutdown. So I have the directory because I need that hook for a clean shutdown with lvm-on-luks. But you, I think, use neither lvm nor encryption right now so you probably haven't included the shutdown hook.

Last edited by cfr (2013-08-04 19:50:48)

WonderWoofy · 2013-08-04 19:58:12

Okay, I figured it out. I don't have that directory when I use the systemd hook. But when I boot with my initramfs that uses base, udev, etc. it shows up just fine.

Edit: And no, I don't use encryption or LVM right now. I just use btrfs. I fine it interesting how much I know about the computers of the regulars on these forums, and how much others know about mine... funny I think.

Oh and BTW, I use the shutdown hook, I have always used the shutdown hook since I used to be a separate var user. But I have since realized that on my simple system, there is no real need for it, and it only make snapshotting (and rolling back) the system with btrfs that much harder.

Last edited by WonderWoofy (2013-08-04 20:00:58)

cfr · 2013-08-04 21:08:04

So did the directory not show up with the shutdown hook and the systemd hook? Or did you mean that you used to use the shutdown hook? I didn't quite understand what you were saying there because you seem to be saying that you use the shutdown hook even though you know it makes snapshotting harder and you don't need it...

Strange if it shows up without the shutdown hook given that the info for that hook seems to suggest that its main point is to copy the initramfs contents to /run/initramfs... but maybe it just means it ensures it is copied there even if it would not otherwise be.

I really wanted to figure out how to use lvm for snapshotting for backups but it seems you need to reboot and I'm afraid I will never backup at all if I have to reboot to do so!

WonderWoofy · 2013-08-04 22:00:52

Yeah rereading that it is not very clear. I no lone ruse a separate var because it makes snapshotting the whole root filesystem more of a PITA. Or more precisely, it makes rolling back the root filesystem more of a PITA when they are separate. This is because with btrfs, I don't specify the subvol= in the fstab, but rather just do it on the kernel command line. So my root filesystem is actually at /rootfs, so I use rootflags=subvol=rootfs on my kernel command line.

I have continuously used the shutdown hook since I noticed that var was suppoedly not being unmounted cleanly at shutdown (the whole logging still running thing). So at that point I added the shutdown hook. It works great, and I think it is a better way of handing things in general.

What makes the /run/initramfs not show up is using the "systemd" mkinitpcio hook that is provided with the systemd 206-1 package in [testing]. This is meant to replace udev, base, usr, and timestamp (well actually the parts of systemd that timestamp added to the initramfs were removed, so it wasn't actually replaced so much as totally deprecated, and systemd in the initramfs happens to provide the same functionality). So I use the shutdown hook for both the "systemd" based initramfs I generate, as well as the "udev" based one that I generate.

So the presence of /run/initramfs has nothing to do with the shutdown hook, that was just my ADD going rampant for a moment there and getting totally off topic.

I don't think that you need to reboot for an LVM snapshot... at least I don't remember it that way at all. LVM2 uses extents, which when snapshotted it simply makes a new root extent that points to the existing extent tree. In reality this should be instantaneous and have nothing to do with rebooting. Though I do remember people having issues with snapshots during boot when there was the switch to the lvmetad way.

cfr · 2013-08-04 22:18:08

Interesting and thanks for the info on lvm. I've been a bit nervous since the bug bit people though I guess it is probably all sorted by now.

I mentioned the shutdown hook, I think, because

$ mkinitcpio -H shutdown
==> Help for hook 'shutdown':
This hook copies the contents of the initramfs into /run/initramfs for reuse
on shutdown. This is needed when you have /usr mounted on a separate partition.

==> This hook has runtime scripts:
  -> cleanup hook

So it is interesting if systemd stops this. Do you get the same information on your system or is this maybe just another difference between a current stable and a testing system? Ie maybe this gets updated with the addition of the systemd hook?

WonderWoofy · 2013-08-04 23:02:59

My thoughts were that if there is the systemd hook in place then the it would be able to properly forward everything in the initramfs to systemd's journal anyway (or maybe forward is the wrong word since it is systemd). I have been using Tom Gunderson's early version of the systemd hook for quite some time now, so I guess I have been getting initramfs logging for a while.

I am still somewhat interested as to why the /run/initramfs directory doesn't exist when I use the systemd hook... whether this is intentional or an oversight.

Anyway, I have properly derailed thsi thread. Back to the topic at hand (which in my mind has been [Solved] over and over again already anyway).

Arch Linux

#1 2013-08-03 14:13:14

Root Partition Warning on Boot

#2 2013-08-03 14:16:39

Re: Root Partition Warning on Boot

#3 2013-08-03 23:39:48

Re: Root Partition Warning on Boot

#4 2013-08-03 23:53:24

Re: Root Partition Warning on Boot

#5 2013-08-04 00:10:57

Re: Root Partition Warning on Boot

#6 2013-08-04 00:33:08

Re: Root Partition Warning on Boot

#7 2013-08-04 00:36:47

Re: Root Partition Warning on Boot

#8 2013-08-04 15:05:17

Re: Root Partition Warning on Boot

#9 2013-08-04 15:26:58

Re: Root Partition Warning on Boot

#10 2013-08-04 15:40:36

Re: Root Partition Warning on Boot

#11 2013-08-04 15:40:39

Re: Root Partition Warning on Boot

#12 2013-08-04 16:15:47

Re: Root Partition Warning on Boot

#13 2013-08-04 16:32:51

Re: Root Partition Warning on Boot

#14 2013-08-04 17:14:35

Re: Root Partition Warning on Boot

#15 2013-08-04 17:26:15

Re: Root Partition Warning on Boot

#16 2013-08-04 17:30:55

Re: Root Partition Warning on Boot

#17 2013-08-04 19:12:59

Re: Root Partition Warning on Boot

#18 2013-08-04 19:18:03

Re: Root Partition Warning on Boot

#19 2013-08-04 19:47:35

Re: Root Partition Warning on Boot

#20 2013-08-04 19:58:12

Re: Root Partition Warning on Boot

#21 2013-08-04 21:08:04

Re: Root Partition Warning on Boot

#22 2013-08-04 22:00:52

Re: Root Partition Warning on Boot

#23 2013-08-04 22:18:08

Re: Root Partition Warning on Boot

#24 2013-08-04 23:02:59

Re: Root Partition Warning on Boot

Board footer