[SOLVED] Btrfs large metadata size

Minsc · 2013-12-03 10:24:13

Hi,

Just converted my /home partition from ext2 to btrfs. First thing I notice is:

 # btrfs fi df /home
Data: total=172.02GB, used=153.14GB
System: total=32.00MB, used=20.00KB
Metadata: total=86.00GB, used=76.41GB

86GB for metadata and 76GB used?? That doesn't look normal - is there any way to reduce that?

Last edited by Minsc (2013-12-04 08:41:22)

graysky · 2013-12-03 10:38:13

Edit. I didn't read your post very carefully. You have quite a bit of data. I forget the ratio of data to metadata. When you conver ext to btrfs, a snapshot of the initial data structure is taken as I recall. I would say, copy the partition elsewhere, format btrfs freshly, and manually copy back.

Last edited by graysky (2013-12-03 10:40:39)

Minsc · 2013-12-03 11:04:08

Yeah I removed the ext2 snapshot - I also tried scrubbing and defragmentation of the partition but the metadata total/usage is still the same.

I would prefer to keep the fresh format as a last resort - was hoping there might be some tuning I could do to decrease the size of the metadata.

Durden · 2013-12-03 14:17:20

Is compression on?
https://wiki.archlinux.org/index.php/Btrfs#Compression

Durden · 2013-12-03 14:19:26

Might be worth doing a scrub also. Is your /home on a RAID or is it just a partition on a single drive? If RAID you might want to do a balance also.

Minsc · 2013-12-03 15:07:32

Hi - compression is disabled at the moment and I've already done a scrub. Not on RAID, but I'm going to give a balance a try tonight anyway to see if that fixes it. If not I will have to go with a clean mkfs.btrfs once I've backed up the data.

I've seen other posts of people's output from "btrfs fi df" and even with terabytes of data total/used the metadata total/used is nowhere near as high as mine. I think this must have something to do with converting from ext2.

Thanks for the replies.

Durden · 2013-12-03 16:05:44

BTW, chances are you'll need to reformat anyway:
https://btrfs.wiki.kernel.org/index.php … lesytem.3F

AdamT · 2013-12-04 02:29:49

Minsc wrote:

Hi - compression is disabled at the moment and I've already done a scrub. Not on RAID, but I'm going to give a balance a try tonight anyway to see if that fixes it. If not I will have to go with a clean mkfs.btrfs once I've backed up the data...

Minsc, I think balance is going to do the trick for you after some cursory reading.

https://wiki.archlinux.org/index.php/Bt … m_Ext3.2F4
"To complete, delete the saved image, delete the sub-volume that image is on, then balance the drive to reclaim the space."

https://btrfs.wiki.kernel.org/index.php … _from_Ext3
"This does create copies of all the Ext3 metadata..."

https://btrfs.wiki.kernel.org/index.php … e.22_do.3F
"If there is a lot of allocated but unused data or metadata chunks, a balance may reclaim some of that allocated space. This is the main reason for running a balance on a single-device filesystem."

https://btrfs.wiki.kernel.org/index.php … s_of_space

https://btrfs.wiki.kernel.org/index.php/Balance_Filters

After balance, consider also converting to skinny extents: btrfstune -x /dev/sdX
http://lkml.indiana.edu/hypermail/linux … 01064.html

Btrfs is awesome, but learn to love https://btrfs.wiki.kernel.org.

Please let me know the results as I have not messed around with conversions.

For reference:

Data, RAID0: total=58.00GB, used=27.17GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=16.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=291.12MB
Metadata: total=8.00MB, used=0.00

Data: total=695.61GB, used=643.16GB
System, DUP: total=8.00MB, used=80.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.50GB, used=810.45MB
Metadata: total=8.00MB, used=0.00

WonderWoofy · 2013-12-04 03:41:21

If you are going to do a balance, make sure your machine is up to date. There has been a bug with balancing and preallocated extents for several weeks, and the patch finally made it to the stable kernel. So you need to be running 3.12.2 or you will get checksum errors and data loss.

Minsc · 2013-12-04 08:41:07

Hi Guys,

Thanks for the replies. I ended up spending the better part of 7 hours last night doing a much needed backup and went with a clean format. It's fixed now.

I would have done a balance, but it would have taken a long time and I needed everything working as normal by this morning.

# df -h /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda9       259G   51G  207G  20% /home

# btrfs fi df /home
Data: total=50.01GB, used=49.68GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=334.64MB
Metadata: total=8.00MB, used=0.00

I moved ~100GB onto another partition so there is a lot less data there, still the metadata usage looks normal now.

AdamT - apologies for the lack of balance results!

AdamT · 2013-12-04 22:09:44

Hey, good to hear that you are up and running as needed Minsc, and no worries!

Do keep in mind skinny extents (and of course, compression) if you find yourself in need of more space.

Take care,

graysky · 2013-12-04 22:35:54

AdamT wrote:

Do keep in mind skinny extents (and of course, compression) if you find yourself in need of more space.

If the CPU is relatively modern, lzo compression actually speeds up I/O throughput plus gives you a space boost.

Arch Linux

#1 2013-12-03 10:24:13

[SOLVED] Btrfs large metadata size

#2 2013-12-03 10:38:13

Re: [SOLVED] Btrfs large metadata size

#3 2013-12-03 11:04:08

Re: [SOLVED] Btrfs large metadata size

#4 2013-12-03 14:17:20

Re: [SOLVED] Btrfs large metadata size

#5 2013-12-03 14:19:26

Re: [SOLVED] Btrfs large metadata size

#6 2013-12-03 15:07:32

Re: [SOLVED] Btrfs large metadata size

#7 2013-12-03 16:05:44

Re: [SOLVED] Btrfs large metadata size

#8 2013-12-04 02:29:49

Re: [SOLVED] Btrfs large metadata size

#9 2013-12-04 03:41:21

Re: [SOLVED] Btrfs large metadata size

#10 2013-12-04 08:41:07

Re: [SOLVED] Btrfs large metadata size

#11 2013-12-04 22:09:44

Re: [SOLVED] Btrfs large metadata size

#12 2013-12-04 22:35:54

Re: [SOLVED] Btrfs large metadata size

Board footer