Btrfs: System won't boot anymore after unclean shutdown

tesfabpel · 2015-04-01 17:50:30

Hi,
I have a problem with my ArchLinux (kernel 3.19.2-1-ARCH).

I have my root partition as BTRFS and no /boot partition.

After an unclean shutdown the system won't boot anymore.
GRUB loads the kernel and the kernel starts but it hangs saying: Mounted /dev/sda3 on real root.

Then, I have to start a Live CD and do:

btrfsck /dev/sda3

After that the system boots correctly.

I've tried adding fsck to the HOOKS in the mkinitcpio.conf file and in fact it tries to fsck the filesystem (but with BTRFS, fsck.btrfs doesn't work...)

What can I do?

Thanks

alphaniner · 2015-04-01 18:11:27

fsck.btrfs(8) wrote:

NAME
fsck.btrfs - do nothing, successfully
...
DESCRIPTION
...
If you wish to check the consistency of a BTRFS filesystem or repair a damaged filesystem, see btrfs-check(8).

Last edited by alphaniner (2015-04-01 18:12:21)

tesfabpel · 2015-04-01 18:17:32

Yes I know.
In fact, I said that I tried to add the fsck HOOK but of course it didn't work because of this "fsck.btrfs - do nothing, successfully"...

The problem is, why the system/kernel "hangs" after saying something like "Mounted /dev/sda3 on real root"?

alphaniner · 2015-04-01 18:25:25

Ok, I got the impression you were unaware that fsck shouldn't work.

ouroboros8 · 2015-04-02 12:06:27

I'm also having this problem. I tried booting up the netinstall image to debug it, but I can't mount my btrfs volume from there either: mount just hangs. After a while, the kernel goes:

[...] INFO: task mount:612 blocked for more than 120 seconds.
[...]          Not tainted 3.19.2-1-ARCH #1
[...] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[...] INFO: task btrfs-transacti:636 blocked for more than 120 seconds.
[...]          Not tainted 3.19.2-1-ARCH #1
[...] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

From this post I guessed that I might just need to wait so that I could defrag the volume once it was mounted, but it's been hanging for over 15min now (the message above continues to come up periodically).

Like the OP in the thread I linked to above, my btrfs volume is inside an LVM volume (which is on dm-crypt). Don't know whether that's relevant.

Output of btrfs-check:

Checking filesystem on /dev/pixie/btrfs
UUID: e04cc661-e31a-4dbf-870d-664bc913e34a
found 76855677062 bytes used err is 1
total csum bytes: 72928436
total tree bytes: 2092515328
total fs tree bytes: 1888452608
total extent tree bytes: 104644608
btree space waste bytes: 357251032
file data blocks allocated: 181197139968
 referenced 109440282624
btrfs-progs v3.19

dmesg output:

[   36.824998] Btrfs loaded
[   36.826713] BTRFS: device fsid e04cc661-e31a-4dbf-870d-664bc913e34a devid 1 transid 98618 /dev/dm-3
[   46.219014] BTRFS info (device dm-3): disk space caching is enabled
[   46.259830] BTRFS: detected SSD devices, enabling SSD mode
[  240.333313] INFO: task mount:539 blocked for more than 120 seconds.
[  240.335542]       Not tainted 3.19.2-1-ARCH #1
[  240.337795] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.340263] mount           D ffff8800cf2837f8     0   539    453 0x00000000
[  240.340277]  ffff8800cf2837f8 ffff8800cf2331b0 0000000000013f00 ffff8800cf283fd8
[  240.340286]  0000000000013f00 ffffffff81818540 ffff8800cf2331b0 ffff8802094f67a8
[  240.340291]  0000000000000004 ffff8800cf283758 ffffffffa0a2fc91 ffff8802094f67a8
[  240.340298] Call Trace:
[  240.340335]  [<ffffffffa0a2fc91>] ? set_extent_buffer_dirty+0x71/0xc0 [btrfs]
[  240.340357]  [<ffffffffa0a05796>] ? btrfs_buffer_uptodate+0x56/0x80 [btrfs]
[  240.340366]  [<ffffffff8155e999>] schedule+0x29/0x70
[  240.340389]  [<ffffffffa0a47745>] btrfs_tree_lock+0x65/0x200 [btrfs]
[  240.340396]  [<ffffffff810b4380>] ? wait_woken+0x90/0x90
[  240.340410]  [<ffffffffa09e5cd9>] btrfs_search_slot+0x799/0xaa0 [btrfs]
[  240.340425]  [<ffffffffa09e79f5>] btrfs_insert_empty_items+0x75/0xd0 [btrfs]
[  240.340446]  [<ffffffffa0a47a37>] btrfs_insert_orphan_item+0x67/0xa0 [btrfs]
[  240.340465]  [<ffffffffa0a4a12e>] insert_orphan_item+0x5e/0x90 [btrfs]
[  240.340482]  [<ffffffffa0a4c531>] replay_one_buffer+0x361/0x380 [btrfs]
[  240.340504]  [<ffffffffa0a2f244>] ? alloc_extent_buffer+0x34/0x370 [btrfs]
[  240.340523]  [<ffffffffa0a488f3>] walk_down_log_tree+0x1d3/0x410 [btrfs]
[  240.340540]  [<ffffffffa0a48bfb>] walk_log_tree+0xcb/0x1d0 [btrfs]
[  240.340558]  [<ffffffffa0a507f7>] btrfs_recover_log_trees+0x1e7/0x490 [btrfs]
[  240.340575]  [<ffffffffa0a4c1d0>] ? replay_one_dir_item+0xf0/0xf0 [btrfs]
[  240.340596]  [<ffffffffa0a08b3e>] open_ctree+0x17ae/0x2030 [btrfs]
[  240.340609]  [<ffffffffa09dd633>] btrfs_mount+0x843/0x920 [btrfs]
[  240.340618]  [<ffffffff8117b7aa>] ? pcpu_alloc+0x35a/0x680
[  240.340628]  [<ffffffff811d5d68>] mount_fs+0x38/0x1c0
[  240.340634]  [<ffffffff8117bb05>] ? __alloc_percpu+0x15/0x20
[  240.340641]  [<ffffffff811f116b>] vfs_kern_mount+0x6b/0x120
[  240.340647]  [<ffffffff811f4184>] do_mount+0x224/0xb40
[  240.340653]  [<ffffffff811f4dae>] SyS_mount+0x7e/0xe0
[  240.340661]  [<ffffffff81562809>] system_call_fastpath+0x12/0x17
[  240.340671] INFO: task btrfs-transacti:561 blocked for more than 120 seconds.
[  240.342927]       Not tainted 3.19.2-1-ARCH #1
[  240.345175] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  240.348473] btrfs-transacti D ffff88020b88bd98     0   561      2 0x00000000
[  240.348483]  ffff88020b88bd98 ffff880211edc590 0000000000013f00 ffff88020b88bfd8
[  240.348491]  0000000000013f00 ffff8802162289f0 ffff880211edc590 ffff8802094a70c8
[  240.348496]  ffff8802094d8000 ffff880200000000 ffff8802094d8000 ffff880200000033
[  240.348501] Call Trace:
[  240.348511]  [<ffffffff811b5d1a>] ? kmem_cache_alloc+0x16a/0x180
[  240.348525]  [<ffffffffa09e07ca>] ? btrfs_free_path+0x2a/0x40 [btrfs]
[  240.348543]  [<ffffffffa0a64213>] ? __btrfs_run_delayed_items+0xf3/0x1d0 [btrfs]
[  240.348550]  [<ffffffff8155e999>] schedule+0x29/0x70
[  240.348568]  [<ffffffffa0a0b555>] btrfs_commit_transaction+0x2a5/0xbe0 [btrfs]
[  240.348573]  [<ffffffff810b4380>] ? wait_woken+0x90/0x90
[  240.348590]  [<ffffffffa0a06fc5>] transaction_kthread+0x1d5/0x240 [btrfs]
[  240.348605]  [<ffffffffa0a06df0>] ? btrfs_cleanup_transaction+0x5a0/0x5a0 [btrfs]
[  240.348611]  [<ffffffff81091748>] kthread+0xd8/0xf0
[  240.348617]  [<ffffffff81091670>] ? kthread_create_on_node+0x1c0/0x1c0
[  240.348624]  [<ffffffff81562758>] ret_from_fork+0x58/0x90
[  240.348629]  [<ffffffff81091670>] ? kthread_create_on_node+0x1c0/0x1c0

Last edited by ouroboros8 (2015-04-02 12:37:25)

nstgc · 2015-04-02 12:56:30

Whenever I have an unclean shutdown (something that has happened quite a few times, twice this week alone infact) is I boot into something that wasn't mounted at the time. I keep an extra arch installation for just such a purpose, as well as a Live arch disk. I first run btrfs check even though it doesn't do anything I like to know it has anything to say before I try to mount the partitions in the volumes to scrub, then run the check again if there were errors since from my experience scrubbing can introduce errors.

If you can't mount try mounting with the options ro,recovery,degraded. You can't scrub with ro, but you also can't do any damage.

ouroboros8 · 2015-04-02 14:50:28

Resolved on my end with the following command.
Back up your btrfs partition before trying this, as it's potentially destructive.

# btrfs check --repair <device>

nstgc · 2015-04-02 17:04:33

In general you should have a robust back up scheme already if using btrfs.

papajoke · 2015-04-02 17:30:15

exactly the same error with my arch (also after an unclean shutdown ) ! ... ?

[240.442749] INFO : task mount:130 blocked for more than 120 second
[240.442828]             not tainted 3.19.2-1-ARCH #1
[240.442895]  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this messages

[480.xxxxxx] INFO : task btrfs-transacti:152 bloked for more than 120 second
[480.xxxxxx]             not tainted 3.19.2-1-ARCH #1
[480.xxxxxx]  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this messages

after btrfs check --repair it's ok

Last edited by papajoke (2015-04-02 17:33:03)

ratcheer · 2015-04-02 19:09:05

I believe the error is a kernel bug. There are several similar bootup failures mentioned in the btrfs mail list: linux-btrfs@vger.kernel.org, Topic: Btrfs hangs 3.19-10

I do not know how to link an email message to a thread post, or I would have done so,

Tim

WorMzy · 2015-04-02 20:00:25

@ratcheer The mailing list archives are available on spinics and gmane (although gmane appears to be suffering from a configuration problem at the moment). Here's the topic you mentioned.

In general, I would recommend adding /usr/bin/btrfs and /usr/bin/btrfs-zero-log (and possibly btrfs-select-super) to your BINARIES array in your mkinitcpio.conf, and creating a boot entry in YourFavouriteBootloader with break=premount in the kernel arguments. That way you can boot into this "rescue" option and have the various btrfs commands available without needing a liveCD.

nstgc · 2015-04-02 22:52:12

WorMzy wrote:

In general, I would recommend adding /usr/bin/btrfs and /usr/bin/btrfs-zero-log (and possibly btrfs-select-super) to your BINARIES array in your mkinitcpio.conf, and creating a boot entry in YourFavouriteBootloader with break=premount in the kernel arguments. That way you can boot into this "rescue" option and have the various btrfs commands available without needing a liveCD.

Not to sound like a total n00b (but I pretty much am), you mean like this:

from /etc/mkinitcpio.conf

# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=" /usr/bin/btrfs /usr/bin/btrfs-zero-log btrfs-select-super"       # <-----

# FILES

AS for the bootloader bit (Grub in my case), do you mean to add the line to "break=premount" when something goes wrong only, or to edit /etc/grub.d/10_linux so as to make this available in /boot/grub/grub.cfg (since we aren't suppose to edit the later, though I have in the past )?

[edit] Sorry, brain fart.

I'd edit /etc/default/grub

GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="break=premount" # <----
GRUB_CMDLINE_LINUX=""

Still, the question stands if I would edit that when I go to boot or if I would edit the configurations to be more permanent.

Last edited by nstgc (2015-04-02 22:56:23)

WorMzy · 2015-04-02 23:02:16

You probably need to specify the full path for btrfs-select-super too, but yes.

As for the bootloader, I'm not sure how you would do it for GRUB2, but in syslinux you would just add a stanza for the recovery option, e.g.

 LABEL archrecovery
         LINUX ../vmlinuz-linux
         APPEND root=/dev/sda2 rw break=premount
         INITRD ../initramfs-linux.img

If you're comfortable modifying your bootloader on the fly, then you'll probably be fine without a dedicated entry, so long as you remember the "break=premount" argument.

EDIT: You don't want to make "break=premount" the default. It's whole purpose is to disrupt the boot process, so you'll be dumped to an initramfs shell.

Last edited by WorMzy (2015-04-02 23:04:09)

nstgc · 2015-04-02 23:10:53

Thank you. As for remembering, I won't, but I do have a laptop. I just have to remember that it exists. In any case, I won't likely need that unless my back up installation also goes belly up.

Arch Linux

#1 2015-04-01 17:50:30

Btrfs: System won't boot anymore after unclean shutdown

#2 2015-04-01 18:11:27

Re: Btrfs: System won't boot anymore after unclean shutdown

#3 2015-04-01 18:17:32

Re: Btrfs: System won't boot anymore after unclean shutdown

#4 2015-04-01 18:25:25

Re: Btrfs: System won't boot anymore after unclean shutdown

#5 2015-04-02 12:06:27

Re: Btrfs: System won't boot anymore after unclean shutdown

#6 2015-04-02 12:56:30

Re: Btrfs: System won't boot anymore after unclean shutdown

#7 2015-04-02 14:50:28

Re: Btrfs: System won't boot anymore after unclean shutdown

#8 2015-04-02 17:04:33

Re: Btrfs: System won't boot anymore after unclean shutdown

#9 2015-04-02 17:30:15

Re: Btrfs: System won't boot anymore after unclean shutdown

#10 2015-04-02 19:09:05

Re: Btrfs: System won't boot anymore after unclean shutdown

#11 2015-04-02 20:00:25

Re: Btrfs: System won't boot anymore after unclean shutdown

#12 2015-04-02 22:52:12

Re: Btrfs: System won't boot anymore after unclean shutdown

#13 2015-04-02 23:02:16

Re: Btrfs: System won't boot anymore after unclean shutdown

#14 2015-04-02 23:10:53

Re: Btrfs: System won't boot anymore after unclean shutdown

Board footer