You are not logged in.

#1 2014-01-18 00:00:46

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

btrfs-convert destroyed my system

I guess this is an issue for the btrfs guys, but I thought I would post here in case anyone else has run into the same issue. To start off, I have an encrypted LVM setup with a root logical volume and a home logical volume. Today decided to upgrade my home LV to btrfs for compression. I installed btrfs-progs, unmounted /home, and ran

btrfs-convert /dev/MyVolumeGroup/home

and it completed with no errors reported. I rebooted my system, and I got a "Welcome to emergency mode!" message. I rebooted into a live CD and found that all of my logical volumes were showing up, but all of them except one (it happened to be a snapshot of the root LV) were showing up as status "NOT available". I ran pvck and lvck, both of which found no errors. I ran lvscan and still /dev/MyVolumeGroup/ (and /dev/mapper/MyVolumeGroup-*) contains only the snapshot, which is useless without the original LV, and no other LVs.

Any ideas about what could cause this, or how I can fix it? It seems that btrfs-convert possibly overwrote the LVM metadata somehow, but I have no idea how since the argument was a logical volume. Even if I had accidentally typed /dev/MyVolumeGroup, I would think that btrfs-convert should have realized that it was not an ext2/3/4 filesystem.

Edit: actually the root LV's status is "available" but mounting it gives

mount: /dev/MyVolumeGroup/root is write-protected, mounting read only.
mount: special device /dev/MyVolumeGroup/root does not exist.

Edit: Possible solution at post #9.

Last edited by EscapedNull (2014-01-18 14:08:20)

Offline

#2 2014-01-18 00:10:53

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: btrfs-convert destroyed my system

Never been a fan of fs conversion tools.  If you have the space, copy the source to a backup media and restore it to a freshly formatted filesystem.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#3 2014-01-18 00:20:31

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

graysky wrote:

Never been a fan of fs conversion tools.

That makes two of us, now.

graysky wrote:

If you have the space, copy the source to a backup media and restore it to a freshly formatted filesystem.

I did have space and that's what I should have done, but it's too late now. Realistically I think I'll have to resort to starting over and creating a new LVM. Nothing on it was irreplaceable, but it's definitely a setback. I'd still like to find out what happened and why though, if only for my own satisfaction.

Offline

#4 2014-01-18 02:17:48

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

Well, the btrfs guys are super responsive.  You should ask on the linux-btrfs mailing list or the #btrfs irc channel.  You should get a pretty quick response, even if it only to say that you may be screwed.

Offline

#5 2014-01-18 03:09:20

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

WonderWoofy wrote:

Well, the btrfs guys are super responsive.  You should ask on the linux-btrfs mailing list or the #btrfs irc channel.  You should get a pretty quick response, even if it only to say that you may be screwed.

I've already started a fresh installation, so I won't be able to provide feedback on the state of my (probably screwed) LVM. I may take your advice anyway though, to at least share my experience with them and perhaps gain some insight.

Offline

#6 2014-01-18 03:10:40

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

Well, if the filesystem is now written over, then there may not be much insight they can actually give you.  But it would probably good for them to know that someone has experienced issues in general with btrfs-convert.

Offline

#7 2014-01-18 04:18:28

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

WonderWoofy wrote:

Well, if the filesystem is now written over, then there may not be much insight they can actually give you.  But it would probably good for them to know that someone has experienced issues in general with btrfs-convert.

Yeah, I realize that, and I wish I could be of more assistance, but I really need to get this system back up and running. I posted on the mailing list so we'll see where that goes. Thanks for the suggestion.

Offline

#8 2014-01-18 04:59:14

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

Yeah, I saw your post.  I subscribe to the mailing list.  We'll see what comes of it.

Offline

#9 2014-01-18 14:06:27

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

Get a load of this. I recreated my LVM this morning with exactly the same LVs, with the same names and sizes, in the same order. When I ran mkfs.btrfs on my home LV, it said "Filesystem already exists. Use -f to override." I mounted it, and my files are there! I guess the LVs were re-created at exactly the same offsets as before. My root LV would probably be there too if I hadn't zeroed the first hundred megabytes or so (which I sometimes do just to make sure old metadata can't somehow get mixed with new. Superstition, maybe, and a poor decision on my part). If I had to choose one, I wish my root LV had been the recoverable one, but at least I'm only half screwed now.

So, if you fall victim to the btrfs-convert on LVM problem, my advice would be:
1) Make a backup!
2) Don't use btrfs-convert!
If you didn't make a backup, and you used btrfs-convert, and your LVM began acting strangely,
3) pvcreate -ff /dev/sdx1
4) vgcreate YourVG /dev/sdx1
Use exactly the same parameters and create them in exactly the same order you did before.
5) lvcreate -n 'YourLV' -L 10G YourVG
5) Try to mount your logical volumes. If you're lucky, your files will be there.

Edit: Oh, and by the way, my home LV is now a valid btrfs filesystem. btrfs-convert worked, it just managed to seriously confuse LVM somehow. Maybe it was as simple as a new UUID causing the confusion, who knows. I still wouldn't trust it.

Last edited by EscapedNull (2014-01-18 14:12:32)

Offline

#10 2014-01-18 14:16:13

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: btrfs-convert destroyed my system

Maybe only your LVM metadata was corrupted, just as you suggested. With a metadata backup, you might have restored it using vgcfgrestore.

Last edited by progandy (2014-01-18 14:22:19)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#11 2014-01-18 15:12:44

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: btrfs-convert destroyed my system

EscapedNull wrote:

Get a load of this. I recreated my LVM this morning with exactly the same LVs, with the same names and sizes, in the same order. When I ran mkfs.btrfs on my home LV, it said "Filesystem already exists. Use -f to override." I mounted it, and my files are there! I guess the LVs were re-created at exactly the same offsets as before. My root LV would probably be there too if I hadn't zeroed the first hundred megabytes or so (which I sometimes do just to make sure old metadata can't somehow get mixed with new. Superstition, maybe, and a poor decision on my part). If I had to choose one, I wish my root LV had been the recoverable one, but at least I'm only half screwed now.

So, if you fall victim to the btrfs-convert on LVM problem, my advice would be:
1) Make a backup!
2) Don't use btrfs-convert!
If you didn't make a backup, and you used btrfs-convert, and your LVM began acting strangely,
3) pvcreate -ff /dev/sdx1
4) vgcreate YourVG /dev/sdx1
Use exactly the same parameters and create them in exactly the same order you did before.
5) lvcreate -n 'YourLV' -L 10G YourVG
5) Try to mount your logical volumes. If you're lucky, your files will be there.

Edit: Oh, and by the way, my home LV is now a valid btrfs filesystem. btrfs-convert worked, it just managed to seriously confuse LVM somehow. Maybe it was as simple as a new UUID causing the confusion, who knows. I still wouldn't trust it.

If I may add my two cents here: Compression on BTRFS is not retroactive; only newly created files are compressed. Converting an existing filesystem will not compress the extant files on the new BTRFS filesystem. If your purpose in converting to BTRFS is its compression feature, then backing up data and reformatting the partition, and copying data back is the right way to go. So you'll now want to move those files off of the filesystem and back on to compress them.

Offline

#12 2014-01-18 15:56:02

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

progandy wrote:

Maybe only your LVM metadata was corrupted, just as you suggested. With a metadata backup, you might have restored it using vgcfgrestore.

You could be right, but I find it unlikely that vgck and lvck wouldn't have detected that, or even lvscan. On the other hand, I have no idea how reliable those tools are. I'm still new to LVM so I didn't realize there was such a thing as LVM metadata backup, but that could have possibly saved me if I'd made a backup and restored it with vgcfgrestore as you said. Luckily this system was still in the early stages and nothing on it was irreplaceable, otherwise I never would have risked an in-place conversion.

ANOKNUSA wrote:
EscapedNull wrote:

Get a load of this. I recreated my LVM this morning with exactly the same LVs, with the same names and sizes, in the same order. When I ran mkfs.btrfs on my home LV, it said "Filesystem already exists. Use -f to override." I mounted it, and my files are there! I guess the LVs were re-created at exactly the same offsets as before. My root LV would probably be there too if I hadn't zeroed the first hundred megabytes or so (which I sometimes do just to make sure old metadata can't somehow get mixed with new. Superstition, maybe, and a poor decision on my part). If I had to choose one, I wish my root LV had been the recoverable one, but at least I'm only half screwed now.

So, if you fall victim to the btrfs-convert on LVM problem, my advice would be:
1) Make a backup!
2) Don't use btrfs-convert!
If you didn't make a backup, and you used btrfs-convert, and your LVM began acting strangely,
3) pvcreate -ff /dev/sdx1
4) vgcreate YourVG /dev/sdx1
Use exactly the same parameters and create them in exactly the same order you did before.
5) lvcreate -n 'YourLV' -L 10G YourVG
5) Try to mount your logical volumes. If you're lucky, your files will be there.

Edit: Oh, and by the way, my home LV is now a valid btrfs filesystem. btrfs-convert worked, it just managed to seriously confuse LVM somehow. Maybe it was as simple as a new UUID causing the confusion, who knows. I still wouldn't trust it.

If I may add my two cents here: Compression on BTRFS is not retroactive; only newly created files are compressed. Converting an existing filesystem will not compress the extant files on the new BTRFS filesystem. If your purpose in converting to BTRFS is its compression feature, then backing up data and reformatting the partition, and copying data back is the right way to go. So you'll now want to move those files off of the filesystem and back on to compress them.

Thanks for the tip ANOKNUSA. I didn't know that. I've done some in-place filesystem conversion and partition moving and things like that before, but I'll admit I was playing with fire every time. Backup-format-restore really is the right way to go. I understand that btrfs-convert only replaces metadata, and the file blocks are left unchanged. Knowing that, it makes perfect sense that existing files would not be compressed. The volume is mostly empty so new files are more important anyway. A simple cp should re-write and compress the file, right?

Offline

#13 2014-01-19 01:50:12

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: btrfs-convert destroyed my system

A simple cp should re-write and compress the file, right?

I'm not sure about that, but you can always give it a try. If that's the case, you should see a measurable difference in free space without having to copy too many files (read: waste any substantial amount of time).  Note also that BTRFS won't compress any file format that already utilizes compression, like .jpg and .mp3. From my (very limited) experience, though, it will still free up a decent amount of space and improve performance on spinning disks, and BTRFS is fundamentally optimized for SSDs. My point (with both this post and the last) is: Don't view the switch to BTRFS and the trouble with your LVM group as wasted time. wink It's a learning experience, wasn't too much trouble to fix and there's some benefit to using BTRFS for every use case. I just felt like mentioning the caveat with compression, since that's what you mentioned was the main selling point.

Offline

#14 2014-01-19 02:37:17

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

Yes copying the file should produce a compressed copy.  But you can actually use the btrfs tool to apply compression as well while you do a defrag.  There is a page that covers this in the btrfs wiki, but its basic usage is also covered in the btrfs man page and the --help.

Offline

#15 2014-01-19 05:11:02

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

ANOKNUSA wrote:

A simple cp should re-write and compress the file, right?

I'm not sure about that, but you can always give it a try. If that's the case, you should see a measurable difference in free space without having to copy too many files (read: waste any substantial amount of time).  Note also that BTRFS won't compress any file format that already utilizes compression, like .jpg and .mp3. From my (very limited) experience, though, it will still free up a decent amount of space and improve performance on spinning disks, and BTRFS is fundamentally optimized for SSDs. My point (with both this post and the last) is: Don't view the switch to BTRFS and the trouble with your LVM group as wasted time. wink It's a learning experience, wasn't too much trouble to fix and there's some benefit to using BTRFS for every use case. I just felt like mentioning the caveat with compression, since that's what you mentioned was the main selling point.

I definitely learned from it. Thankfully it didn't take too long to set everything up again, either. Even if the compression isn't perfect, it's still much better than what you get with ext4 (which is none at all in most cases).

WonderWoofy wrote:

Yes copying the file should produce a compressed copy.  But you can actually use the btrfs tool to apply compression as well while you do a defrag.  There is a page that covers this in the btrfs wiki, but its basic usage is also covered in the btrfs man page and the --help.

Thanks for the suggestion, but I think I'm going to stay away from those types of tools for a while.

Offline

#16 2014-01-19 05:33:32

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

But those tools are what makes btrfs awesome!  In any case, I would recommend mounting with the 'autodefrag' mount option if you don't mind a little bit of write amplification.  Being a CoW filesystem, defragmentation can get pretty bad over time with btrfs.  That mount flag isn't without issues though, so just beware if you decide you want to start using quotas or qgroups.  If you stay away from those, 'autodefrag' is fine.  Fixes are being work on by josef as well.

Offline

#17 2014-01-19 21:45:00

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

WonderWoofy wrote:

But those tools are what makes btrfs awesome!  In any case, I would recommend mounting with the 'autodefrag' mount option if you don't mind a little bit of write amplification.  Being a CoW filesystem, defragmentation can get pretty bad over time with btrfs.  That mount flag isn't without issues though, so just beware if you decide you want to start using quotas or qgroups.  If you stay away from those, 'autodefrag' is fine.  Fixes are being work on by josef as well.

Well, yes. I probably won't be able to avoid those tools forever. I've just had bad experiences with them so far (which is what this thread is about, after all). I'll look into autodefrag. I just need to sit down and do some reading about btrfs; there's quite a bit more administration involved with it than with ext4.

Offline

#18 2014-01-19 22:27:04

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: btrfs-convert destroyed my system

EscapedNull wrote:

I find it unlikely that vgck and lvck wouldn't have detected that, or even lvscan.

It wouldn't, if the change was intentional.

Although I don't see why btrfs-convert would want to change the LV arrangement without asking...

You're really lucky if recreating gave you LVs with the same layout, that does not usually happen (different order in which you create the LVs, or different sizes, or ...).

It's safer to just use the backups (/etc/lvm/*) even if you have to dig through filesystem raw data to find them (which is harder with encryption, probably impossible if you add LV fragmentation).

Never been a fan of fs conversion tools.

If you don't have a backup, your data wasn't important. smile

That said, just because there are some bad tools, shouldn't give such tools a bad name in general. Filesystem conversion is absolutely possible. Especially with LVM in the mix, you should be able to snapshot the filesystem, convert, and roll back if it did not work. Of course that only works as long as the tool does not intentionally break the LVM in the first place. That's just silly of the tool if it really does that.

Last edited by frostschutz (2014-01-19 22:29:01)

Offline

#19 2014-01-19 22:29:35

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

That is because ext4 doesn't do nearly as many cool things as btrfs smile

From reading the linux-btrfs mailing list, it appears as though the 'btrfs-convert' tool does not seem to get as much love and attention as the primary 'btrfs' command.  The eventual goal is to get everything into that once central tool, though I am not sure if btrfs-convert is part of that.  So things that are not included in the 'btrfs' command tend to be things that are either still being worked on, or things that are really for diagnostic/development/rescue purposes only.

As a new btrfs user, you should be aware that btrfsck (or 'btrfs check') should not be used all willy-nilly like other fsck tools.  If you ever run into problems where you think you may need to use it, it is best to ask on the btrfs mailing list or #btrfs first.

Offline

#20 2014-01-19 22:39:11

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: btrfs-convert destroyed my system

Even for ext* there are cases where fsck causes more damage than it repairs; if you have a badly damaged filesystem, with really important & not backed up data on, you should always make a full image or snapshot first, before experimenting with it.

I don't use btrfs for the simple reason it's still considered experimental... (and it really doesn't offer anything I need, happy with traditional filesystems on LVM)

Offline

#21 2014-01-20 02:22:04

EscapedNull
Member
Registered: 2013-12-04
Posts: 129

Re: btrfs-convert destroyed my system

frostschutz wrote:
EscapedNull wrote:

I find it unlikely that vgck and lvck wouldn't have detected that, or even lvscan.

It wouldn't, if the change was intentional.

To what degree do you define "intentional"? I didn't intend for most of my LVs to become "NOT available".

frostschutz wrote:

Although I don't see why btrfs-convert would want to change the LV arrangement without asking...

You're really lucky if recreating gave you LVs with the same layout, that does not usually happen (different order in which you create the LVs, or different sizes, or ...).

I was quite surprised by that too.

frostschutz wrote:

It's safer to just use the backups (/etc/lvm/*) even if you have to dig through filesystem raw data to find them (which is harder with encryption, probably impossible if you add LV fragmentation).

Never been a fan of fs conversion tools.

That said, just because there are some bad tools, shouldn't give such tools a bad name in general. Filesystem conversion is absolutely possible. Especially with LVM in the mix, you should be able to snapshot the filesystem, convert, and roll back if it did not work. Of course that only works as long as the tool does not intentionally break the LVM in the first place. That's just silly of the tool if it really does that.

That would be the case if the LVM itself was intact, but somehow something during the btrfs-convert process caused most of my LVs to become unavailable and unmountable. I can't explain it, but btrfs-convert managed to damage the LVM itself despite being given an LV as an argument. Even if I had given the device (in this case a LUKS container) itself as the argument, 1) Why wasn't the LUKS container broken, 2) Why didn't btrfs-convert realize there was no ext* filesystem there, and 3) Why did it successfully convert my /home LV to a working btrfs filesystem?

frostschutz wrote:

If you don't have a backup, your data wasn't important. smile

Backups are good, but I didn't post here because I expected someone to get my data back. I posted here to warn other users about my experience with btrfs-convert, and possibly find out how it could have happened. Thus, "you should have made a backup" isn't exactly the answer I was looking for.

WonderWoofy wrote:

That is because ext4 doesn't do nearly as many cool things as btrfs smile

From reading the linux-btrfs mailing list, it appears as though the 'btrfs-convert' tool does not seem to get as much love and attention as the primary 'btrfs' command.  The eventual goal is to get everything into that once central tool, though I am not sure if btrfs-convert is part of that.  So things that are not included in the 'btrfs' command tend to be things that are either still being worked on, or things that are really for diagnostic/development/rescue purposes only.

I guess I should have paid closer attention to the warning label before using btrfs-convert, but I didn't realize it was less supported than the main "btrfs" command. I'm almost afraid to use btrfs at all now.

WonderWoofy wrote:

As a new btrfs user, you should be aware that btrfsck (or 'btrfs check') should not be used all willy-nilly like other fsck tools.  If you ever run into problems where you think you may need to use it, it is best to ask on the btrfs mailing list or #btrfs first.

Well then I'll be btrfscked. But thanks once again for the tip.

Offline

#22 2014-01-20 19:28:36

davidm
Member
Registered: 2009-04-25
Posts: 371

Re: btrfs-convert destroyed my system

I don't know.  I did recently have a multi-device btrfs filesytem (data=single, metadata=raid1) become corrupt though.  It happened due to a power supply failure (under-volting) and I had 192 errors and about 25 recoverable errors (presumably meta-data) after a scrub.  I have backups for the important data but somehow I thought it would fare better in this sort of situation even without using raid1 for the data.  I'm thinking I might move to ZFS (with a raid1 type set up this time!) for a while to give btrfs time to mature.

There are many mixed messages given about btrfs from many sources.  Some sources are acting as if it is as stable as ext4 but it's not and one look at the mailing list should show that.  From what I read you're USUALLY fine if you stay away from the outlier cases and as long as nothing goes wrong but if it does you might be using those backups.

Last edited by davidm (2014-01-20 19:29:43)

Offline

#23 2014-01-20 23:50:35

WonderWoofy
Member
From: Los Gatos, CA
Registered: 2012-05-19
Posts: 8,414

Re: btrfs-convert destroyed my system

It is NOT as stable as ext4. 

But for a filesystem in such heavy development it is quite reliable for the majority of users.  You are correct in saying that you should try to stay away from the outlier cases, as these tend to be the people who find their way to the mailing list for borked systems.  Typically this involves those who have thousands upon thousands of subvolumes and snapshots, a gazillion hard links (for which btrfs is probably not the ideal filesystem actually), or try to use the newest and least developed/tested features. 

But for the day to day desktop user, it can be an amazing filesystem.  I would still recommend keeping tested backups... though shouldn't you already do this anyway?  Oh and don't keep all your backups on btrfs either (though I do for my laptop, but all my important data is also kept on a home file server as well).

Offline

Board footer

Powered by FluxBB