btrfs forced readonly [SOLVED]

brentaar · 2014-10-25 01:25:07

Last night the root btrfs on my main box became borked. It became read only and on a restart non-existent. after trying to troubleshoot the issue, I ended up paving over and starting fresh.

Now with a fresh install I'm seeing the same type of issues, thought system will boot into gnome, and after a bit it'll be forced to read only.

I ran smartctl on the disk and the root disk passed. (There is one disk that did not though is is just a storage disk)

What are the correct trouble shooting steps?

thank you.

i have the output dmesg

[ 2223.735916] ------------[ cut here ]------------
[ 2223.735956] WARNING: CPU: 11 PID: 5620 at fs/btrfs/extent-tree.c:876 btrfs_lookup_extent_info+0x471/0x4c0 [btrfs]()
[ 2223.735959] Modules linked in: xfs libcrc32c jfs bnep bluetooth crc16 fuse arc4 snd_hda_codec_hdmi ath9k nls_iso8859_1 ath9k_common nls_cp437 vfat ath9k_hw fat ath mousedev coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul mac80211 hid_generic crc32_pclmul ghash_clmulni_intel eeepc_wmi iTCO_wdt asus_wmi snd_usb_audio sparse_keymap iTCO_vendor_support led_class aesni_intel video hwmon evdev aes_x86_64 lrw usbhid mac_hid mxm_wmi snd_usbmidi_lib gf128mul snd_rawmidi glue_helper psmouse ablk_helper hid cryptd sb_edac serio_raw cfg80211 snd_seq_device pcspkr i2c_i801 edac_core rtc_efi nvidia(PO) rfkill r8169 mii lpc_ich snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep drm snd_pcm snd_timer snd tpm_infineon mei_me i2c_core
[ 2223.736008]  tpm_tis mei soundcore tpm button processor wmi shpchp nfs lockd sunrpc fscache btrfs xor raid6_pq sd_mod sr_mod crc_t10dif cdrom crct10dif_common atkbd libps2 crc32c_intel ahci libahci libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i8042 serio
[ 2223.736034] CPU: 11 PID: 5620 Comm: snapperd Tainted: P           O   3.17.1-1-ARCH #1
[ 2223.736036] Hardware name: System manufacturer System Product Name/P9X79 LE, BIOS 4608 12/24/2013
[ 2223.736039]  0000000000000000 00000000d5e94126 ffff880ec18ab6b0 ffffffff81536850
[ 2223.736042]  0000000000000000 ffff880ec18ab6e8 ffffffff8107054d ffff880f031ad240
[ 2223.736046]  ffff8800cdb51000 000000001dd8c000 ffff880f66ceb320 0000000000000000
[ 2223.736049] Call Trace:
[ 2223.736057]  [<ffffffff81536850>] dump_stack+0x4d/0x6f
[ 2223.736064]  [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0
[ 2223.736068]  [<ffffffff8107067a>] warn_slowpath_null+0x1a/0x20
[ 2223.736083]  [<ffffffffa04bac71>] btrfs_lookup_extent_info+0x471/0x4c0 [btrfs]
[ 2223.736099]  [<ffffffffa04c1fd1>] ? btrfs_alloc_free_block+0x3b1/0x480 [btrfs]
[ 2223.736112]  [<ffffffffa04ab31c>] update_ref_for_cow+0x18c/0x370 [btrfs]
[ 2223.736131]  [<ffffffffa04f9750>] ? copy_extent_buffer+0xb0/0x110 [btrfs]
[ 2223.736144]  [<ffffffffa04ab72c>] __btrfs_cow_block+0x22c/0x530 [btrfs]
[ 2223.736157]  [<ffffffffa04abbc6>] btrfs_cow_block+0x116/0x1e0 [btrfs]
[ 2223.736170]  [<ffffffffa04af9b3>] btrfs_search_slot+0x1e3/0xa70 [btrfs]
[ 2223.736187]  [<ffffffffa04dedaf>] btrfs_truncate_inode_items+0x12f/0xa70 [btrfs]
[ 2223.736192]  [<ffffffff811c9c38>] ? __sb_start_write+0x58/0x110
[ 2223.736208]  [<ffffffffa04e0928>] btrfs_evict_inode+0x3e8/0x560 [btrfs]
[ 2223.736216]  [<ffffffff811e24d4>] evict+0xb4/0x1a0
[ 2223.736219]  [<ffffffff811e2d65>] iput+0xf5/0x1a0
[ 2223.736233]  [<ffffffffa04e1548>] btrfs_orphan_cleanup+0x208/0x440 [btrfs]
[ 2223.736248]  [<ffffffffa04e1b80>] btrfs_lookup_dentry+0x400/0x5a0 [btrfs]
[ 2223.736262]  [<ffffffffa04e1d32>] btrfs_lookup+0x12/0x40 [btrfs]
[ 2223.736267]  [<ffffffff811d0b6d>] lookup_real+0x1d/0x70
[ 2223.736272]  [<ffffffff811d1872>] __lookup_hash+0x42/0x60
[ 2223.736275]  [<ffffffff811d2333>] lookup_slow+0x43/0xc0
[ 2223.736279]  [<ffffffff811d44d3>] path_lookupat+0x723/0x880
[ 2223.736282]  [<ffffffff811d268c>] ? getname_flags+0x2c/0x130
[ 2223.736286]  [<ffffffff811d4656>] filename_lookup.isra.30+0x26/0x80
[ 2223.736290]  [<ffffffff811d7967>] user_path_at_empty+0x67/0xd0
[ 2223.736293]  [<ffffffff811d79e1>] user_path_at+0x11/0x20
[ 2223.736297]  [<ffffffff811cc34a>] vfs_fstatat+0x6a/0xd0
[ 2223.736301]  [<ffffffff811c90aa>] ? __fput+0x13a/0x200
[ 2223.736305]  [<ffffffff811cca69>] SyS_newfstatat+0x29/0x60
[ 2223.736310]  [<ffffffff8108c684>] ? task_work_run+0xa4/0xe0
[ 2223.736315]  [<ffffffff81014d45>] ? do_notify_resume+0x95/0xa0
[ 2223.736320]  [<ffffffff8153ca60>] ? int_signal+0x12/0x17
[ 2223.736324]  [<ffffffff8153c829>] system_call_fastpath+0x16/0x1b
[ 2223.736327] ---[ end trace b1b5980b59c9b9cb ]---
[ 2223.736331] BTRFS: error (device sda3) in update_ref_for_cow:1018: errno=-30 Readonly filesystem
[ 2223.736334] BTRFS info (device sda3): forced readonly
[ 2223.736336] ------------[ cut here ]------------
[ 2223.736347] WARNING: CPU: 11 PID: 5620 at fs/btrfs/super.c:259 __btrfs_abort_transaction+0x50/0x110 [btrfs]()
[ 2223.736349] BTRFS: Transaction aborted (error -30)
[ 2223.736350] Modules linked in: xfs libcrc32c jfs bnep bluetooth crc16 fuse arc4 snd_hda_codec_hdmi ath9k nls_iso8859_1 ath9k_common nls_cp437 vfat ath9k_hw fat ath mousedev coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul mac80211 hid_generic crc32_pclmul ghash_clmulni_intel eeepc_wmi iTCO_wdt asus_wmi snd_usb_audio sparse_keymap iTCO_vendor_support led_class aesni_intel video hwmon evdev aes_x86_64 lrw usbhid mac_hid mxm_wmi snd_usbmidi_lib gf128mul snd_rawmidi glue_helper psmouse ablk_helper hid cryptd sb_edac serio_raw cfg80211 snd_seq_device pcspkr i2c_i801 edac_core rtc_efi nvidia(PO) rfkill r8169 mii lpc_ich snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep drm snd_pcm snd_timer snd tpm_infineon mei_me i2c_core
[ 2223.736397]  tpm_tis mei soundcore tpm button processor wmi shpchp nfs lockd sunrpc fscache btrfs xor raid6_pq sd_mod sr_mod crc_t10dif cdrom crct10dif_common atkbd libps2 crc32c_intel ahci libahci libata ehci_pci xhci_hcd ehci_hcd scsi_mod usbcore usb_common i8042 serio
[ 2223.736418] CPU: 11 PID: 5620 Comm: snapperd Tainted: P        W  O   3.17.1-1-ARCH #1
[ 2223.736420] Hardware name: System manufacturer System Product Name/P9X79 LE, BIOS 4608 12/24/2013
[ 2223.736422]  0000000000000000 00000000d5e94126 ffff880ec18ab730 ffffffff81536850
[ 2223.736425]  ffff880ec18ab778 ffff880ec18ab768 ffffffff8107054d 00000000ffffffe2
[ 2223.736428]  ffff880f66ceb320 ffff8800cdb51000 ffffffffa0552220 0000000000000488
[ 2223.736432] Call Trace:
[ 2223.736435]  [<ffffffff81536850>] dump_stack+0x4d/0x6f
[ 2223.736439]  [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0
[ 2223.736445]  [<ffffffff810705cc>] warn_slowpath_fmt+0x5c/0x80
[ 2223.736459]  [<ffffffffa04ab4e1>] ? update_ref_for_cow+0x351/0x370 [btrfs]
[ 2223.736471]  [<ffffffffa04a5d10>] __btrfs_abort_transaction+0x50/0x110 [btrfs]
[ 2223.736484]  [<ffffffffa04ab91d>] __btrfs_cow_block+0x41d/0x530 [btrfs]
[ 2223.736497]  [<ffffffffa04abbc6>] btrfs_cow_block+0x116/0x1e0 [btrfs]
[ 2223.736509]  [<ffffffffa04af9b3>] btrfs_search_slot+0x1e3/0xa70 [btrfs]
[ 2223.736524]  [<ffffffffa04dedaf>] btrfs_truncate_inode_items+0x12f/0xa70 [btrfs]
[ 2223.736528]  [<ffffffff811c9c38>] ? __sb_start_write+0x58/0x110
[ 2223.736542]  [<ffffffffa04e0928>] btrfs_evict_inode+0x3e8/0x560 [btrfs]
[ 2223.736549]  [<ffffffff811e24d4>] evict+0xb4/0x1a0
[ 2223.736552]  [<ffffffff811e2d65>] iput+0xf5/0x1a0
[ 2223.736566]  [<ffffffffa04e1548>] btrfs_orphan_cleanup+0x208/0x440 [btrfs]
[ 2223.736579]  [<ffffffffa04e1b80>] btrfs_lookup_dentry+0x400/0x5a0 [btrfs]
[ 2223.736593]  [<ffffffffa04e1d32>] btrfs_lookup+0x12/0x40 [btrfs]
[ 2223.736598]  [<ffffffff811d0b6d>] lookup_real+0x1d/0x70
[ 2223.736602]  [<ffffffff811d1872>] __lookup_hash+0x42/0x60
[ 2223.736605]  [<ffffffff811d2333>] lookup_slow+0x43/0xc0
[ 2223.736609]  [<ffffffff811d44d3>] path_lookupat+0x723/0x880
[ 2223.736612]  [<ffffffff811d268c>] ? getname_flags+0x2c/0x130
[ 2223.736615]  [<ffffffff811d4656>] filename_lookup.isra.30+0x26/0x80
[ 2223.736619]  [<ffffffff811d7967>] user_path_at_empty+0x67/0xd0
[ 2223.736622]  [<ffffffff811d79e1>] user_path_at+0x11/0x20
[ 2223.736626]  [<ffffffff811cc34a>] vfs_fstatat+0x6a/0xd0
[ 2223.736629]  [<ffffffff811c90aa>] ? __fput+0x13a/0x200
[ 2223.736633]  [<ffffffff811cca69>] SyS_newfstatat+0x29/0x60
[ 2223.736638]  [<ffffffff8108c684>] ? task_work_run+0xa4/0xe0
[ 2223.736642]  [<ffffffff81014d45>] ? do_notify_resume+0x95/0xa0
[ 2223.736646]  [<ffffffff8153ca60>] ? int_signal+0x12/0x17
[ 2223.736650]  [<ffffffff8153c829>] system_call_fastpath+0x16/0x1b
[ 2223.736652] ---[ end trace b1b5980b59c9b9cc ]---
[ 2223.736672] BTRFS error (device sda3): Error removing orphan entry, stopping orphan cleanup
[ 2223.736675] BTRFS critical (device sda3): could not do orphan cleanup -22
[ 2223.736865] systemd-journald[242]: Failed to truncate file to its own size: Read-only file system
[ 2223.736924] systemd-journald[242]: Failed to truncate file to its own size: Read-only file system
[ 2223.738099] systemd-journald[242]: Failed to truncate file to its own size: Read-only file system
[ 2223.738182] systemd-journald[242]: Failed to truncate file to its own size: Read-only file system
...repeated...

Last edited by brentaar (2014-10-25 17:15:57)

brentaar · 2014-10-25 01:38:33

Running btrfs check /dev/sda3 gives a lot of "backpointer mismatch" errors.

To be clear, this system was just created this morning, and should have no issues.

I'm running the long smartctl test to see if anything turns up.

ratcheer · 2014-10-25 02:10:46

What kernel are you running? btrfs has some severe bugs in 3.17.0 and 3.17.1. Corruption is one of the problems. I am hoping the patches will be in 3.17.2.

Tim

brentaar · 2014-10-25 02:52:54

This box is running 3.17.1-1.

davidm · 2014-10-25 05:24:14

ratcheer wrote:

What kernel are you running? btrfs has some severe bugs in 3.17.0 and 3.17.1. Corruption is one of the problems. I am hoping the patches will be in 3.17.2.
Tim

I thought I read the corruption issues are mainly involving systems with snapshots though. Or is it affecting even those without snapshots?

brentaar · 2014-10-25 05:33:54

I'm still catching up on the issue myself, so I can't speak to those.

My issue lies with a running disk.

some more specifics

uefi
gummiboot

/boot - sda1 - fat32
swap - sda2
/ - sda3 - mount a subvolume from the root volume with a flag set in gummiboot, btrfs
linux kernel 3.17.1

2 other btrfs volumes don't seem to be affected (different side note one drive seems to have SMART issues)

I took a dd of the borked disk last night before nuking and i took a dd of the fresh installed disk before i noticed issues.

I have been using snapper to take snapshots, but I have not been using the snapshots.

So my problem is with a subvol and not snapshots.

Last edited by brentaar (2014-10-25 06:57:06)

brentaar · 2014-10-25 17:13:54

After jumping online in the #btrfs room, this issue has been cleared up.

It seems that it was the same issue as ratcheer mentioned

ratcheer wrote:

What kernel are you running? btrfs has some severe bugs in 3.17.0 and 3.17.1. Corruption is one of the problems. I am hoping the patches will be in 3.17.2.
Tim

Snapper was creating snapshots and i was using the 3.17.1 kernel.

After getting the newest version of the btrfs-progs from git and compiling them.
I ran btrfs check /dev/sda3 and btrfs check --repair /dev/sda3.
And removed snapper.

I left the system on over night and it is working.

I'm surprised that this bug was let out into the wild with how bad it was.

Now I'll try to recover the other disk that was borked (a dd image), hopefully I won't loose what's on it.

ratcheer · 2014-10-31 00:44:33

Ouch! It seems that the necessary btrfs patches, which were submitted just before the 3.17.1 kernel was released, did not make it into 3.17.2, either.

Tim

eigengrau · 2014-11-02 17:03:26

As per #btrfs, the fix should be included in 3.17.2.

Edit: Indeed, it’s committed as 0bd7729 in the 3.17.2 changelog.

Last edited by eigengrau (2014-11-02 17:05:05)

thiagowfx · 2014-11-02 20:42:09

Does upgrading to 3.17.2-1 fix the problem? I mean, after recovering the broken / corrupt filesystem as brentaar did.

ratcheer · 2014-11-02 23:37:09

thiagowfx wrote:

Does upgrading to 3.17.2-1 fix the problem? I mean, after recovering the broken / corrupt filesystem as brentaar did.

Yes, apparently. I have upgraded from 3.16.4 to 3.17.2 and so far, my system is working just fine.

Tim

thiagowfx · 2014-11-04 14:41:10

I can confirm it works.

I wiped my / (btrfs), reinstalled Arch from scratch on it, then upgraded the linux package, installed btrfs-progs-git and rebooted. After that, I did a btrfsck on my home partition (also btrfs), and the output told me to run it again, this time with the --repair option. After a minute or so, my /home was successfully repaired.

The only downside of this history was that I couldn't recover my root (I wiped it, as I said). Why? I was trying to do the same procedure (compiling btrfs-progs-git and running btrfsck on my root), however, it wasn't sufficient to have the latest version of btrfs-progs; it looks like you also had to be running on a recent Linux kernel (3.16+). A person from #btrfs also confirmed this to me.

Problem was...I couldn't think of any linux distribution who has a live medium of a recent Linux kernel. I used grml (which I've found to be a very nice recover distro), however its linux were 3.12. Okay, of course I also tried with Arch, which were on 3.17.1, however I always got a problem related with BUFFER I/O and ext4 error -- I believe I was stressing my USB Flash Drive way too much by installing several packages on it (required to compile then install the btrfs-progs-git AUR package).

TL;DR: all is well, except that I'm redownloading and reinstalling all of my packages.

Last edited by thiagowfx (2014-11-04 14:41:32)

nstgc · 2014-11-04 19:18:52

Both Ubuntu and Fedora RC I believe use 3.16. Obviously that doesn't help now.

thiagowfx · 2014-11-05 22:50:28

Ubuntu 14.04 came with 3.12.x, at least its initial ISO. I haven't checked Fedora RC, but it is a good idea to keep it in mind for a next trouble =P

Arch Linux

#1 2014-10-25 01:25:07

btrfs forced readonly [SOLVED]

#2 2014-10-25 01:38:33

Re: btrfs forced readonly [SOLVED]

#3 2014-10-25 02:10:46

Re: btrfs forced readonly [SOLVED]

#4 2014-10-25 02:52:54

Re: btrfs forced readonly [SOLVED]

#5 2014-10-25 05:24:14

Re: btrfs forced readonly [SOLVED]

#6 2014-10-25 05:33:54

Re: btrfs forced readonly [SOLVED]

#7 2014-10-25 17:13:54

Re: btrfs forced readonly [SOLVED]

#8 2014-10-31 00:44:33

Re: btrfs forced readonly [SOLVED]

#9 2014-11-02 17:03:26

Re: btrfs forced readonly [SOLVED]

#10 2014-11-02 20:42:09

Re: btrfs forced readonly [SOLVED]

#11 2014-11-02 23:37:09

Re: btrfs forced readonly [SOLVED]

#12 2014-11-04 14:41:10

Re: btrfs forced readonly [SOLVED]

#13 2014-11-04 19:18:52

Re: btrfs forced readonly [SOLVED]

#14 2014-11-05 22:50:28

Re: btrfs forced readonly [SOLVED]

Board footer