BTRFS root and home partition suddenly not able to mount

falken · 2019-11-25 04:32:15

Hey everyone. I'm running an up to date arch install on a uefi system with one SSD and one 1tb mechanical hard drive. I had the SSD in 3 partitions, boot, swap, and the root and home partititon as btrfs (sdb4).

I'm intending this system to be used as an offline gaming machine for a friend so I had been downloading and installing a lot of games on the system using wine, lutris etc. The first time this happened, the SSD was nearly full. I rebooted and arch started again and I was greeted with "wrong fs type, bad option, bad superblock on /dev/sdb4, missing codepage or helper program, or other error" and was dropped into an emergency shell. I thought at this point when I was formatting and mounting the 1 tb mechanical hard drive that I messed something up on the SSD, I thought the SSD was corrupted so I reformatted and reinstalled arch.

On the 2nd install I kept everything the same, 3 partitions on the SSD, with home & root being btrfs, and then swap and boot partitions. I also had the 1tb mechanical hard drive formatted as ext4. Everything was working fine until the SSD started to become full. I noticed there were about 14 gb free on the SSD ( was installing and testing GTA V at the time) when I rebooted and was greeted with the same boot error I got last time, "wrong fs type, bad option, bad superblock on /dev/sdb4, missing codepage or helper program, or other error.", and dropped into an emergency shell

I tried googling around for solutions but i'm not able to mount /dev/sdb4 from either the emergency shell or a live usb enviorment. Trying to mount in a live usb enviorment says

BTRFS error (device sdb4): failed to recover balance: -5
BTRFS error (device sdb4): open_ctree failed
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb4, missing codepage or helper program, or other error

Also, I should add that I tried btrfs rescue super-recover /dev/sdb4 and got:

All supers are valid, no need to recover

I tried things like mount o,recovery, mount recovery, etc. I should note that every other device mounts just fine except for the btrfs dev/sdb4.

I'm kind of at a loss here. I think the data is fine because this last error happened at the exact same partition with the exact same file system with extremely similar cirucmstances.

If anyone has any help with this I would greatly appreciate it.

Here's the link to the accompanying reddit post I made incase any info there helps anyone. https://www.reddit.com/r/archlinux/comm … _troubles/

d_fajardo · 2019-11-25 10:50:43

Have you checked the filesystem with btrfs check <device> or btrfs check --repair? You can run a live environment and run btrfs check. The good thing is it needs to be unmounted to check.

falken · 2019-11-25 11:08:00

d_fajardo wrote:

Have you checked the filesystem with btrfs check <device> or btrfs check --repair? You can run a live environment and run btrfs check. The good thing is it needs to be unmounted to check.

Hey d_fajardo. I just tried that right now, with this output;

enabling repair mode
Opening filesystem to check...
Checking filesystem on /dev/sdb4
UUID: 02b4f414-6e02-4342-b7e3-f3e3ed3fa8c3
[1/7] checking root items
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
Csum didn't match
ERROR: failed to repair root items: Input/output error

I was able to atleast use btrfs restore to my secondary hard drive, it shows the files being there (if they're all OK i'll have to figure out) so I might be able to just format, but I would like to know if its possible to salvage this. It is quite an old SSD this is on so let me know if you need the output of smartctl

Last edited by falken (2019-11-25 11:09:20)

d_fajardo · 2019-11-25 12:55:32

I'm not entirely clear. Did you run the check on your SSD or sbd4? With your SSD, I assume your boot is in FAT32 so you can just fsck that. I assume your root is in btrfs so you can run btrfs check on that.
While you're at it, you might as well run smartctl on your drives and see if they pass the simplest tests.

kermit63 · 2019-11-25 14:45:08

Did you by any chance resize /dev/sdb4 after its initial creation? I'm not talking about resizing the volumes within /dev/sdb4 but resizing /dev/sdb4 itself.

falken · 2019-11-25 17:17:27

d_fajardo wrote:

I'm not entirely clear. Did you run the check on your SSD or sbd4? With your SSD, I assume your boot is in FAT32 so you can just fsck that. I assume your root is in btrfs so you can run btrfs check on that.
While you're at it, you might as well run smartctl on your drives and see if they pass the simplest tests.

I just ran it on sbd4. The other 2 partitions are not btrfs and boot/mount just fine. I will run that later today and post what I get

falken · 2019-11-25 17:18:03

kermit63 wrote:

Did you by any chance resize /dev/sdb4 after its initial creation? I'm not talking about resizing the volumes within /dev/sdb4 but resizing /dev/sdb4 itself.

Nope, I did not. It was created as about a 214 GiB partition and I didnt alter that at all

falken · 2019-11-25 19:22:36

falken wrote:

d_fajardo wrote:
I'm not entirely clear. Did you run the check on your SSD or sbd4? With your SSD, I assume your boot is in FAT32 so you can just fsck that. I assume your root is in btrfs so you can run btrfs check on that.
While you're at it, you might as well run smartctl on your drives and see if they pass the simplest tests.
I just ran it on sbd4. The other 2 partitions are not btrfs and boot/mount just fine. I will run that later today and post what I get

https://imgur.com/a/fQEaptm

Here is the output of smartctl -a on all devices and partitions. Apologies for the low quality picture, I dont have any other way of getting it out

d_fajardo · 2019-11-25 19:33:08

What is the output of # smartctl -H /dev/<device>? Refer to this.

ugjka · 2019-11-25 19:52:31

btrfsck --repair --init-extent-tree
btrfsch --repair --init-csum-tree

These can be destructive, make sure to make a back up your btrfs partition
and don't interrupt the repair process with Ctrl+C or somehow else

falken · 2019-11-25 20:01:25

d_fajardo wrote:

What is the output of # smartctl -H /dev/<device>? Refer to this.

# smartctl -H /dev/sdb4

SMART overall-health self-assessment test result: PASSED

d_fajardo · 2019-11-25 21:37:08

You can try ugjka's suggestion but make sure you clone the drive first as suggested.

Slithery · 2019-11-25 21:40:44

falken wrote:

Apologies for the low quality picture, I dont have any other way of getting it out

https://wiki.archlinux.org/index.php/Li … in_clients

falken · 2019-11-25 22:33:05

ugjka wrote:

btrfsck --repair --init-extent-tree
btrfsch --repair --init-csum-tree
These can be destructive, make sure to make a back up your btrfs partition
and don't interrupt the repair process with Ctrl+C or somehow else

I backed up before doing these commands to my 1tb mechanical hdd. Here's the output of those commands.

# btrfsck --repair --init-extent-tree /dev/sdb4

enabling repair mode
Opening file system to check...
Checking filesystem on /dev/sdb4
UUID: 02b4f414-6e02-4342-b7e3-f3e3ed3fa8c3
Creating a new extent tree
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
Csum didn't match
Error reading tree block
error pinning down used bytes
ERROR: attempt to start transcation over already running one
f extent buffer leak: start 788267008 len 16384

# btrfs check --repair --init-csum-tree /dev/sdb4

enabling repair mode
Creating a new CRC tree
Opening file system to check...
Checking filesystem on /dev/sdb4
UUID: 02b4f414-6e02-4342-b7e3-f3e3ed3fa8c3
Reintialize checksum tree
[1/7] checking root items
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
checksum verify failed on 685342720 found C32A031C wanted 24B94F9A
Csum didn't match
ERROR: failed to repair root items: Input/output error
btrfs check --repair -init-csum-tree /dev/sdb4 639.44s user 93.20s system 84% cpu 14:29.46 total

d_fajardo · 2019-11-26 15:33:54

It looks to me the filesystem is not repairable. What puzzles me is why you are getting the error on your 1TB HDD when your SSD is the one getting full. I don't have an answer. It seems there's a threshold on your SSD that triggers something to the other drive when reached.
My setup is similar to you but slightly different. I have a small drive with 4 partitions - for the root, swap, boot, and home, and a large 2TB data exclusively for my files and data. My /home partition is understandably very small on the small drive but what I do is just symlink the different directories of my data drive to my /home. I do this so that whatever happens to the system drive, my data drive remains intact. Also I can just symlink directories I am currently working on and deleting them when I know I'm not going to touch them for a while. I'm not sure if it's going to work for your setup or your needs but you can give it a try like that if you intend to redo everything.

Ropid · 2019-11-26 16:37:49

falken wrote:

falken wrote:
d_fajardo wrote:
I'm not entirely clear. Did you run the check on your SSD or sbd4? With your SSD, I assume your boot is in FAT32 so you can just fsck that. I assume your root is in btrfs so you can run btrfs check on that.
While you're at it, you might as well run smartctl on your drives and see if they pass the simplest tests.
I just ran it on sbd4. The other 2 partitions are not btrfs and boot/mount just fine. I will run that later today and post what I get
https://imgur.com/a/fQEaptm
Here is the output of smartctl -a on all devices and partitions. Apologies for the low quality picture, I dont have any other way of getting it out

If I read that table right, there was one sector that's broken on your drive. The drive "reallocated" it, meaning it replaced it with one of its hidden extra sectors.

Is that table in your photo the drive with your broken filesystem? Maybe the drive's hardware error happened recently and is the reason for the problem?

falken · 2019-11-26 17:26:05

d_fajardo wrote:

It looks to me the filesystem is not repairable. What puzzles me is why you are getting the error on your 1TB HDD when your SSD is the one getting full. I don't have an answer. It seems there's a threshold on your SSD that triggers something to the other drive when reached.
My setup is similar to you but slightly different. I have a small drive with 4 partitions - for the root, swap, boot, and home, and a large 2TB data exclusively for my files and data. My /home partition is understandably very small on the small drive but what I do is just symlink the different directories of my data drive to my /home. I do this so that whatever happens to the system drive, my data drive remains intact. Also I can just symlink directories I am currently working on and deleting them when I know I'm not going to touch them for a while. I'm not sure if it's going to work for your setup or your needs but you can give it a try like that if you intend to redo everything.

I'm not getting the error on the HDD, I got it on the ssd. /dev/sdb/ is my SSD

d_fajardo · 2019-11-26 17:57:47

Ah ok so I misunderstood. Question is your SSD in sdb as well? Because if it is why do you have your 1TB in sdb (sdb4) as well? Surely it should be in sda or sdc as it is separate physical drive. Perhaps you had an initial partitioning with btrfs that extended your SSD to your HDD (as one continues drive) before you installed linux?
I would go back and erase all your SSD partitions clean perhaps by memory clearing, redo the partitions and reinstall. Make sure that your 1TB HDD is recognized as another device from your SSD (as another /dev/sdx).

falken · 2019-11-26 18:25:02

d_fajardo wrote:

Ah ok so I misunderstood. Question is your SSD in sdb as well? Because if it is why do you have your 1TB in sdb (sdb4) as well? Surely it should be in sda or sdc as it is separate physical drive. Perhaps you had an initial partitioning with btrfs that extended your SSD to your HDD (as one continues drive) before you installed linux?
I would go back and erase all your SSD partitions clean perhaps by memory clearing, redo the partitions and reinstall. Make sure that your 1TB HDD is recognized as another device from your SSD (as another /dev/sdx).

Sorry I might not have been clear -- my HDD is /dev/sda. They're completely seperate partitions. My ssd is /sdb/, with the 3 aforementioned partitions for boot, swap and root.

Sda is just 1 partition formatted as ext4

d_fajardo · 2019-11-26 18:37:24

Yes my initial comment was wrong anyway because you did reformat your HDD to ext4 and still got the error. If you only have 3 partitions on sdb, how come you have sdb4? I would think it would only be sdb1=boot, sdb2=swap. sdb3=root. What's in the 'missing' partition and what's the fs on that? That partition is what's causing the problem.
Anyway if you are inclined to reinstall, then wipe the drive clean with wipefs or the memory clearing I suggested before reinstall. As this is an old SSD perhaps the memory clearing could boost its performance on reinstall.

cmurf · 2019-12-31 08:09:35

So this is a month old now but...I strongly suspect a hardware problem.

From the smartctl output, it looks like it's saying 28TB lifetime writes. That's quite a lot. But I'm not seeing much of anything else that suggests what's going on. What I can say is that Btrfs is very intolerant of errors: everything is checksummed, fs metadata and data. That's strictly for error detection. Error correction can only happen if there's extra copies of the metadata. By default SSDs have single copy metadata, generally it's not considered worth the extra wear and unknown locality of the second copy anyway, but as a troubleshooting measure it might be useful to use DUP metadata in such a case.

Another thought is to make sure the SSD firmware is up to date. The older the SSD, decently likely there are firmware updates available to fix bugs. The checkcum mismatch isn't a single bit flip like what you'd see with bad RAM. So I suspect the SSD might be flaky.

What mount options are being used for this btrfs? And also is 'fstrim.timer' enabled? I'd advise against using either discard mount option or fstrim.timer while trying to isolate the cause of this problem, if you choose to keep using this hardware.

Arch Linux

#1 2019-11-25 04:32:15

BTRFS root and home partition suddenly not able to mount

#2 2019-11-25 10:50:43

Re: BTRFS root and home partition suddenly not able to mount

#3 2019-11-25 11:08:00

Re: BTRFS root and home partition suddenly not able to mount

#4 2019-11-25 12:55:32

Re: BTRFS root and home partition suddenly not able to mount

#5 2019-11-25 14:45:08

Re: BTRFS root and home partition suddenly not able to mount

#6 2019-11-25 17:17:27

Re: BTRFS root and home partition suddenly not able to mount

#7 2019-11-25 17:18:03

Re: BTRFS root and home partition suddenly not able to mount

#8 2019-11-25 19:22:36

Re: BTRFS root and home partition suddenly not able to mount

#9 2019-11-25 19:33:08

Re: BTRFS root and home partition suddenly not able to mount

#10 2019-11-25 19:52:31

Re: BTRFS root and home partition suddenly not able to mount

#11 2019-11-25 20:01:25

Re: BTRFS root and home partition suddenly not able to mount

#12 2019-11-25 21:37:08

Re: BTRFS root and home partition suddenly not able to mount

#13 2019-11-25 21:40:44

Re: BTRFS root and home partition suddenly not able to mount

#14 2019-11-25 22:33:05

Re: BTRFS root and home partition suddenly not able to mount

#15 2019-11-26 15:33:54

Re: BTRFS root and home partition suddenly not able to mount

#16 2019-11-26 16:37:49

Re: BTRFS root and home partition suddenly not able to mount

#17 2019-11-26 17:26:05

Re: BTRFS root and home partition suddenly not able to mount

#18 2019-11-26 17:57:47

Re: BTRFS root and home partition suddenly not able to mount

#19 2019-11-26 18:25:02

Re: BTRFS root and home partition suddenly not able to mount

#20 2019-11-26 18:37:24

Re: BTRFS root and home partition suddenly not able to mount

#21 2019-12-31 08:09:35

Re: BTRFS root and home partition suddenly not able to mount

Board footer