BTRFS file system corrupted [SOLVED]

jerryDaBaaws · 2025-08-21 08:26:36

My disk was running low on space and during some IO operation , I think it got filled completely for some time. I freed up data , around ~100gb again, but my btrfs system thinks it has no space.

> df -i

> df -h

Filesystem      Size  Used Avail Use% Mounted on
dev             7.3G     0  7.3G   0% /dev
run             7.3G   18M  7.3G   1% /run
efivarfs        246K  134K  108K  56% /sys/firmware/efi/efivars
/dev/nvme1n1p2  724G  412G  307G  58% /
tmpfs           7.3G   14M  7.3G   1% /dev/shm
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs           7.3G  9.6M  7.3G   1% /tmp
/dev/nvme1n1p2  724G  412G  307G  58% /home
/dev/nvme0n1p5  672G  412G  226G  65% /mnt/16dc8b64-0ab0-4357-89c6-958303e4439c
/dev/nvme1n1p1  511M  296K  511M   1% /efi
tmpfs           1.5G  6.3M  1.5G   1% /run/user/1000

> df -i

Filesystem       Inodes  IUsed    IFree IUse% Mounted on
dev             1888354    691  1887663    1% /dev
run             1894374   1203  1893171    1% /run
efivarfs              0      0        0     - /sys/firmware/efi/efivars
/dev/nvme1n1p2        0      0        0     - /
tmpfs           1894374     59  1894315    1% /dev/shm
tmpfs              1024      1     1023    1% /run/credentials/systemd-journald.service
tmpfs           1048576    499  1048077    1% /tmp
/dev/nvme1n1p2        0      0        0     - /home
/dev/nvme0n1p5 44802048 700643 44101405    2% /mnt/16dc8b64-0ab0-4357-89c6-958303e4439c
/dev/nvme1n1p1        0      0        0     - /efi
tmpfs            378874    262   378612    1% /run/user/1000

> sudo btrfs device stats /home

[sudo] password for kushy: 
[/dev/nvme1n1p2].write_io_errs    0
[/dev/nvme1n1p2].read_io_errs     0
[/dev/nvme1n1p2].flush_io_errs    0
[/dev/nvme1n1p2].corruption_errs  0
[/dev/nvme1n1p2].generation_errs  0
[/dev/nvme0n1p3].write_io_errs    0
[/dev/nvme0n1p3].read_io_errs     0
[/dev/nvme0n1p3].flush_io_errs    0
[/dev/nvme0n1p3].corruption_errs  0
[/dev/nvme0n1p3].generation_errs  0

> sudo btrfs filesystem df /home

Data, single: total=391.00GiB, used=389.38GiB
System, DUP: total=32.00MiB, used=80.00KiB
Metadata, DUP: total=14.00GiB, used=10.76GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

please give advice on how to fix it

Last edited by jerryDaBaaws (2025-08-22 13:25:56)

mpan · 2025-08-21 09:27:43

You didn’t describe, how your filesystem is corrupted. What we also miss is information on the layout: the output of `df` suggests that a single filesystem is mounted in two places.

Note: `df` is giving only estimates. For btrfs they’re notoriously so poor, `df` is hardly useful on that filesystem. Use either `btrfs filesystem df` for an overview or `btrfs fi usage` for a more detailed view. Still rough estimates, but much more accurate.

Last edited by mpan (2025-08-21 09:30:26)

jerryDaBaaws · 2025-08-21 10:15:57

> `df` suggests that a single filesystem is mounted in two places.

yes, / and /home, it has been that way since an year.

> You didn’t describe, how your filesystem is corrupted.

i can only speculate as to what led to it, which is basically disk getting entirely filled. What i think is that btrfs metatada partition got filled, and now it can not make changes because it can update metadata. basically a deadlock

> btrfs fi usage /home

[sudo] password for kushy: 
Overall:
    Device size:		723.76GiB
    Device allocated:		415.06GiB
    Device unallocated:		308.69GiB
    Device missing:		    0.00B
    Device slack:		    0.00B
    Used:			411.34GiB
    Free (estimated):		309.87GiB	(min: 155.52GiB)
    Free (statfs, df):		309.87GiB
    Data ratio:			     1.00
    Metadata ratio:		     2.00
    Global reserve:		512.00MiB	(used: 0.00B)
    Multiple profiles:		       no

Data,single: Size:391.00GiB, Used:389.82GiB (99.70%)
   /dev/nvme0n1p2	310.00GiB
   /dev/nvme1n1p3	 81.00GiB

Metadata,DUP: Size:12.00GiB, Used:10.76GiB (89.68%)
   /dev/nvme0n1p2	 10.00GiB
   /dev/nvme1n1p3	 14.00GiB

System,DUP: Size:32.00MiB, Used:96.00KiB (0.29%)
   /dev/nvme0n1p2	 64.00MiB

Unallocated:
   /dev/nvme0n1p2	156.38GiB
   /dev/nvme1n1p3	152.32GiB

> btrfs fi usage /

Overall:
    Device size:		723.76GiB
    Device allocated:		415.06GiB
    Device unallocated:		308.69GiB
    Device missing:		    0.00B
    Device slack:		    0.00B
    Used:			411.34GiB
    Free (estimated):		309.87GiB	(min: 155.52GiB)
    Free (statfs, df):		309.87GiB
    Data ratio:			     1.00
    Metadata ratio:		     2.00
    Global reserve:		512.00MiB	(used: 0.00B)
    Multiple profiles:		       no

Data,single: Size:391.00GiB, Used:389.82GiB (99.70%)
   /dev/nvme0n1p2	310.00GiB
   /dev/nvme1n1p3	 81.00GiB

Metadata,DUP: Size:12.00GiB, Used:10.76GiB (89.68%)
   /dev/nvme0n1p2	 10.00GiB
   /dev/nvme1n1p3	 14.00GiB

System,DUP: Size:32.00MiB, Used:96.00KiB (0.29%)
   /dev/nvme0n1p2	 64.00MiB

Unallocated:
   /dev/nvme0n1p2	156.38GiB
   /dev/nvme1n1p3	152.32GiB

jerryDaBaaws · 2025-08-21 10:16:54

overall size is 720gb is because i had to add another partition to btrfs system so i can atleast boot properly

mpan · 2025-08-21 11:10:00

Ok, but what errors are you getting? Please don’t paraphrase; write the exact error message. Also show, how is the mounting being done for both “/” and “/home”.

The filesystem might benefit from balancing or removal of some forgotten copies of files. But do neither while it’s corrupted, as any writes will only make the situation worse and can lead to data loss.

Last edited by mpan (2025-08-21 11:11:08)

jerryDaBaaws · 2025-08-21 15:11:51

# /dev/nvme0n1p2
UUID=f675181c-0a75-4724-9171-3e9755828bd6	/home     	btrfs     	rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/@home	    0 0

# /dev/nvme0n1p1
UUID=91C7-EADF      	/efi      	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	    0 2

before adding another disk, i was getting "no space left on device" by every program on running from terminal as they were crashing.

also if you see output of df -i, inode of entire disk are 0

mpan · 2025-08-21 19:45:07

Oh, then it seems there is no corruption. Just no space left.

Run balance:

sudo btrfs balance start -dusage=XX /home

Start with `-dusage=0`, as this is almost zero-cost and generally can’t fail. You may then try with `-dusage=10`. Theoretically you may go up to `-dusage=100`, but with hundreds of GB it’s likely to be a waste of time for very little benefit.

If `btrfs balance` itself fails due to no space, temporarily add a device to the filesystem (a gigabyte should be sufficient), balance, and then remove that device.

Also make sure you don’t have any snapshots you don’t need anymore. Btrfs is a copy-on-write filesystem, so removing a file that is also in a snapshot isn’t going to free up any space.

Sorry for the confusion. I followed the initial description of the filesystem being broken, in which case any modifications would need to be avoided and the original cause had to be determined. Cheers.

Last edited by mpan (2025-08-21 19:46:02)

jerryDaBaaws · 2025-08-22 03:52:34

thanks, i did run a balance job, and seems to be fine, btw do you think inodes showing as 0 for root partition is fine ?
in output of df -i

mpan · 2025-08-22 08:06:54

Yes, it’s normal to see 0 there.

Filesystems like ZFS or Btrfs have no fixed upper limit of inodes. They allocate metadata for them as needed. The current inode count is also not well defined or easy to determine. So calls to statvfs leave the related fields with no meaningful value.

Compare fs/btrfs/super.c (lines from 1709 in v6.16):

static int btrfs_statfs(struct dentry *dentry, struct kstatfs *buf)
{
	struct btrfs_fs_info *fs_info = btrfs_sb(dentry->d_sb);
	struct btrfs_super_block *disk_super = fs_info->super_copy;
	struct btrfs_space_info *found;
	u64 total_used = 0;
	u64 total_free_data = 0;
	u64 total_free_meta = 0;
	u32 bits = fs_info->sectorsize_bits;
	__be32 *fsid = (__be32 *)fs_info->fs_devices->fsid;
	unsigned factor = 1;
	struct btrfs_block_rsv *block_rsv = &fs_info->global_block_rsv;
	int ret;
	u64 thresh = 0;
	int mixed = 0;

	list_for_each_entry(found, &fs_info->space_info, list) {
		if (found->flags & BTRFS_BLOCK_GROUP_DATA) {
			int i;

			total_free_data += found->disk_total - found->disk_used;
			total_free_data -=
				btrfs_account_ro_block_groups_free_space(found);

			for (i = 0; i < BTRFS_NR_RAID_TYPES; i++) {
				if (!list_empty(&found->block_groups[i]))
					factor = btrfs_bg_type_to_factor(
						btrfs_raid_array[i].bg_flag);
			}
		}

		/*
		 * Metadata in mixed block group profiles are accounted in data
		 */
		if (!mixed && found->flags & BTRFS_BLOCK_GROUP_METADATA) {
			if (found->flags & BTRFS_BLOCK_GROUP_DATA)
				mixed = 1;
			else
				total_free_meta += found->disk_total -
					found->disk_used;
		}

		total_used += found->disk_used;
	}

	buf->f_blocks = div_u64(btrfs_super_total_bytes(disk_super), factor);
	buf->f_blocks >>= bits;
	buf->f_bfree = buf->f_blocks - (div_u64(total_used, factor) >> bits);

	/* Account global block reserve as used, it's in logical size already */
	spin_lock(&block_rsv->lock);
	/* Mixed block groups accounting is not byte-accurate, avoid overflow */
	if (buf->f_bfree >= block_rsv->size >> bits)
		buf->f_bfree -= block_rsv->size >> bits;
	else
		buf->f_bfree = 0;
	spin_unlock(&block_rsv->lock);

	buf->f_bavail = div_u64(total_free_data, factor);
	ret = btrfs_calc_avail_data_space(fs_info, &total_free_data);
	if (ret)
		return ret;
	buf->f_bavail += div_u64(total_free_data, factor);
	buf->f_bavail = buf->f_bavail >> bits;

	/*
	 * We calculate the remaining metadata space minus global reserve. If
	 * this is (supposedly) smaller than zero, there's no space. But this
	 * does not hold in practice, the exhausted state happens where's still
	 * some positive delta. So we apply some guesswork and compare the
	 * delta to a 4M threshold.  (Practically observed delta was ~2M.)
	 *
	 * We probably cannot calculate the exact threshold value because this
	 * depends on the internal reservations requested by various
	 * operations, so some operations that consume a few metadata will
	 * succeed even if the Avail is zero. But this is better than the other
	 * way around.
	 */
	thresh = SZ_4M;

	/*
	 * We only want to claim there's no available space if we can no longer
	 * allocate chunks for our metadata profile and our global reserve will
	 * not fit in the free metadata space.  If we aren't ->full then we
	 * still can allocate chunks and thus are fine using the currently
	 * calculated f_bavail.
	 */
	if (!mixed && block_rsv->space_info->full &&
	    (total_free_meta < thresh || total_free_meta - thresh < block_rsv->size))
		buf->f_bavail = 0;

	buf->f_type = BTRFS_SUPER_MAGIC;
	buf->f_bsize = fs_info->sectorsize;
	buf->f_namelen = BTRFS_NAME_LEN;

	/* We treat it as constant endianness (it doesn't matter _which_)
	   because we want the fsid to come out the same whether mounted
	   on a big-endian or little-endian host */
	buf->f_fsid.val[0] = be32_to_cpu(fsid[0]) ^ be32_to_cpu(fsid[2]);
	buf->f_fsid.val[1] = be32_to_cpu(fsid[1]) ^ be32_to_cpu(fsid[3]);
	/* Mask in the root object ID too, to disambiguate subvols */
	buf->f_fsid.val[0] ^= btrfs_root_id(BTRFS_I(d_inode(dentry))->root) >> 32;
	buf->f_fsid.val[1] ^= btrfs_root_id(BTRFS_I(d_inode(dentry))->root);

	return 0;
}

… with fs/ext4/super.c (lines from 6850 in v6.16):

static int ext4_statfs(struct dentry *dentry, struct kstatfs *buf)
{
	struct super_block *sb = dentry->d_sb;
	struct ext4_sb_info *sbi = EXT4_SB(sb);
	struct ext4_super_block *es = sbi->s_es;
	ext4_fsblk_t overhead = 0, resv_blocks;
	s64 bfree;
	resv_blocks = EXT4_C2B(sbi, atomic64_read(&sbi->s_resv_clusters));

	if (!test_opt(sb, MINIX_DF))
		overhead = sbi->s_overhead;

	buf->f_type = EXT4_SUPER_MAGIC;
	buf->f_bsize = sb->s_blocksize;
	buf->f_blocks = ext4_blocks_count(es) - EXT4_C2B(sbi, overhead);
	bfree = percpu_counter_sum_positive(&sbi->s_freeclusters_counter) -
		percpu_counter_sum_positive(&sbi->s_dirtyclusters_counter);
	/* prevent underflow in case that few free space is available */
	buf->f_bfree = EXT4_C2B(sbi, max_t(s64, bfree, 0));
	buf->f_bavail = buf->f_bfree -
			(ext4_r_blocks_count(es) + resv_blocks);
	if (buf->f_bfree < (ext4_r_blocks_count(es) + resv_blocks))
		buf->f_bavail = 0;
	buf->f_files = le32_to_cpu(es->s_inodes_count);
	buf->f_ffree = percpu_counter_sum_positive(&sbi->s_freeinodes_counter);
	buf->f_namelen = EXT4_NAME_LEN;
	buf->f_fsid = uuid_to_fsid(es->s_uuid);

#ifdef CONFIG_QUOTA
	if (ext4_test_inode_flag(dentry->d_inode, EXT4_INODE_PROJINHERIT) &&
	    sb_has_quota_limits_enabled(sb, PRJQUOTA))
		ext4_statfs_project(sb, EXT4_I(dentry->d_inode)->i_projid, buf);
#endif
	return 0;
}

Last edited by mpan (2025-08-22 09:14:52)

jerryDaBaaws · 2025-08-22 13:23:04

so its still not fixed, the problem started when i was creating some docker container build, space got filled and this happend.
So i tried deleting /var/lib/docker folder, which i could not do before because of btrfs errors

after adding another partition and rebalancing, i tried deleting the folder again, using sudo, it got stuck
i changed the permission using chmod and deleted it, which worked.

Now i reinstalled docker , and when i try to pull images, they cant access /var/lib/docker folder anymore
> open /var/lib/docker/tmp/GetImageBlob2031697050: no such file or directory

> tree /var/lib/docker
/var/lib/docker [error opening dir]

0 directories, 0 files

I even deleted the folder again, and ran docker again, didnt fix anything

jerryDaBaaws · 2025-08-22 13:25:35

nvm, restarting docker socket worked

thanks a lot

mpan · 2025-08-22 13:44:13

Deleting an entry in a filesystem is only removing the name. The file data and related metadata remain in place for as long as something links to that file. For example through a file description or a page mapping. I suppose this is what has happened.

Arch Linux

#1 2025-08-21 08:26:36

BTRFS file system corrupted [SOLVED]

#2 2025-08-21 09:27:43

Re: BTRFS file system corrupted [SOLVED]

#3 2025-08-21 10:15:57

Re: BTRFS file system corrupted [SOLVED]

#4 2025-08-21 10:16:54

Re: BTRFS file system corrupted [SOLVED]

#5 2025-08-21 11:10:00

Re: BTRFS file system corrupted [SOLVED]

#6 2025-08-21 15:11:51

Re: BTRFS file system corrupted [SOLVED]

#7 2025-08-21 19:45:07

Re: BTRFS file system corrupted [SOLVED]

#8 2025-08-22 03:52:34

Re: BTRFS file system corrupted [SOLVED]

#9 2025-08-22 08:06:54

Re: BTRFS file system corrupted [SOLVED]

#10 2025-08-22 13:23:04

Re: BTRFS file system corrupted [SOLVED]

#11 2025-08-22 13:25:35

Re: BTRFS file system corrupted [SOLVED]

#12 2025-08-22 13:44:13

Re: BTRFS file system corrupted [SOLVED]

Board footer