Btrfs segfault while balancing newly added drive

firecat53 · 2015-05-23 22:13:02

tl;dr: segfault during btrfs RAID 1 balance when adding new drive. Now cancelling the balance is stuck and can't be killed.

Yesterday I had a drive failure (well...I broke the SATA power plug on the drive) in a 2x1TB disk btrfs RAID 1 array. Rebooted and mounted the array with '-o degraded'. I thought at first I could use the 'btrfs device delete missing' but then realized I actually had to _add_ a device first, even though the broken hard drive was already removed. Added a 2TB disk to the array (just to get the remaining 1TB drive mirrored again), started a 'btrfs balance start /home'. It ran for several hours, then got a segfault (systemd journal here. Segfault at the end).

This morning I tried a 'btrfs balance cancel /home' and it's been stuck for over 8 hours now without cancelling. There's nothing in htop or glances that's obvious except the load average (i3 CPU) has been sitting at around 2.0 (normally .1-.3) since the attempted cancel.

All the other services running on the server seem to be operating normally. Backups are up-to-date! I'm a little afraid to reboot for fear of borking the existing 1Tb drive, but I'm not sure what else to try at this point. Any ideas?

Thanks!
Scott

firecat53 · 2015-05-24 18:40:36

Ok, I'm about done with btrfs. I finally rebooted after getting no responses in #btrfs or on the mailing list. Now the remaining drive is unmountable with an 'open_ctree failed' error. I'd love to help troubleshoot this but I kind of need the server back operating.

It's really @#$% annoying too, because now I either have to risk a new btrfs install again or find/implement a different filesystem for my /var/lib/docker directory that plays nicely with Docker.

</rant> sorry...

WorMzy · 2015-05-24 19:12:13

You say that you posted to the mailing list, but which one? I can't see any recent messages about this bug on the btrfs mailing list, but there is a thread about it dating back to late 2014: http://article.gmane.org/gmane.comp.fil … tion+c+242

It doesn't seem like the cause was ever determined. If you've not zapped the filesystem yet, you may be able to pick up where Tomasz left off and help debug the problem. If not, your backstory + BUG message may help identify the cause of the problem. It seems like a really nasty, but rare bug (only two occurrences in five months), so the more information they get about it will be helpful.

EDIT: I should probably warn you about rants too. I understand your frustrations, but please leave these out of your posts, thanks. https://wiki.archlinux.org/index.php/Fo … cs.2Frants

Last edited by WorMzy (2015-05-24 19:16:57)

firecat53 · 2015-05-24 19:47:35

Had to repost the mail I sent yesterday...it must've gotten blocked or greylisted

Scott

Arch Linux

#1 2015-05-23 22:13:02

Btrfs segfault while balancing newly added drive

#2 2015-05-24 18:40:36

Re: Btrfs segfault while balancing newly added drive

#3 2015-05-24 19:12:13

Re: Btrfs segfault while balancing newly added drive

#4 2015-05-24 19:47:35

Re: Btrfs segfault while balancing newly added drive

Board footer