[SOLVED] Unable to update files with pacman, system thinks disk's full

HumungusFungus · 2023-03-20 23:13:53

So I tried running

pacman -Syu

to update my system, when I got an error that there wasn't enough space on my disk left. I checked with

df -h

and it told me 98% of my root directory was full, with only 9.6G available, but I thought that this was clearly wrong, so I ran

du -sh *

and it seems like nothing is wrong, there are no large files at all. Nothing in /var, pacman cache was empty since I ran

pacman -Scc

, and only 182G found in my home directory even though my disk has 477G size. I looked online and tried using

lsof | grep deleted

to see if there were any running processes still using deleted files, which would account for the inflated df -h reading, but no, they were all small processes, using at most 50M, and nowhere near enough of them to account for the almost 300G missing space.

Output for the important commands:
df -h: https://0x0.st/H-lM.txt
du -sh *: https://0x0.st/H-0w.txt (used from / )
lsof | grep deleted: https://0x0.st/H-lB.txt (warning: it's quite long but does not contain any extremely large files)

Is there any explanation or solution for where almost 300G of space has gone?

Last edited by HumungusFungus (2023-03-21 14:34:38)

WorMzy · 2023-03-20 23:25:53

/. snapshots is missing from the du output.

HumungusFungus · 2023-03-20 23:46:42

WorMzy wrote:

/. snapshots is missing from the du output.

Fixed (it's empty). I added the last line with

du .snapshots

called from /

Last edited by HumungusFungus (2023-03-20 23:52:16)

3beb6e7c46a615a · 2023-03-21 07:40:54

Did you run du as root? I'm asking because /root appears to be empty, even though normally there are at least some RC files inside.

Incidentally, you may find an interactive tool like dua easier than du. The GitHub project page has binaries for download in case you currently can't use pacman.

Do you use btrfs? If so do you have other subvolumes not mounted currently?

HumungusFungus · 2023-03-21 09:27:25

lunaryorn wrote:

Did you run du as root? I'm asking because /root appears to be empty, even though normally there are at least some RC files inside.

Oops sorry I forgot, I updated the link to reflect the new output as root.

Incidentally, you may find an interactive tool like dua easier than du. The GitHub project page has binaries for download in case you currently can't use pacman.

Okay, I'll give dua a try. Okay I've installed it, and it's really useful, similar to another tool I have called dust which is like a visual dua, but that one isn't interactive like this. However, it doesn't bring any new information to the table, stating that the total disk usage is 250.17GB, and that 194.42GB (77%) of it comes from my home directory. But it still doesn't come close to showing where the other 200GB of my 477GB disk is.

Do you use btrfs? If so do you have other subvolumes not mounted currently?

Yes I use btrfs, however it seems like I have mounted all of my subvolumes. When I run

btrfs subvolume list -p /

as root, I get this: https://0x0.st/H-GH.txt, of which @, @home, @log, @pkg and @.snapshots are all mounted (the rest look like snapshots made by timeshift as backups), as can be seen when I run

mount | grep subvol

which outputs

/dev/mapper/luksdev on / type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@)
/dev/mapper/luksdev on /.snapshots type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=260,subvol=/@.snapshots)
/dev/mapper/luksdev on /home type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@home)
/dev/mapper/luksdev on /var/cache/pacman/pkg type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=259,subvol=/@pkg)
/dev/mapper/luksdev on /var/log type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=258,subvol=/@log)

(although if you'd like to see the entirety of the mount output without the grep in case it would be helpful that is also here: https://0x0.st/H-GH.txt)

3beb6e7c46a615a · 2023-03-21 10:05:25

(the rest look like snapshots made by timeshift as backups),

So you do not have all your subvolumes mounted… you have approximately 20 extra subvolumes that haven't appeared anywhere yet. These snapshots do require disk space, too, and that's likely where your missing 200 GB have gone. By the way, what's "@home/humfung/Snapshots_BTRFS/root2"? This doesn't look like timeshift at all…

Try to temporarily mount the root subvolume (id 5 I think) to e.g. "/mnt" with "mount -Osubvolid=5 /dev/mapper/luksdev /tmp", and then run "btrfs filesystem du -s /mnt/*" to find out which snapshots hold a large amount of exclusive data. You may want to delete those, and then perhaps look into adjusting timeshift's setting to somehow prevent or cleanup overly large snapshots.

HumungusFungus · 2023-03-21 14:33:14

lunaryorn wrote:

(the rest look like snapshots made by timeshift as backups),
So you do not have all your subvolumes mounted… you have approximately 20 extra subvolumes that haven't appeared anywhere yet. These snapshots do require disk space, too, and that's likely where your missing 200 GB have gone. By the way, what's "@home/humfung/Snapshots_BTRFS/root2"? This doesn't look like timeshift at all…

Oh, yeah that's not timeshift that was just me learning how to use btrfs snapshots, and honestly I kind of forgot it existed so I just deleted it.

Try to temporarily mount the root subvolume (id 5 I think) to e.g. "/mnt" with "mount -Osubvolid=5 /dev/mapper/luksdev /tmp", and then run "btrfs filesystem du -s /mnt/*" to find out which snapshots hold a large amount of exclusive data. You may want to delete those, and then perhaps look into adjusting timeshift's setting to somehow prevent or cleanup overly large snapshots.

So I tried this but for some reason it was taking a very long time to calculate the space, but just to be safe I deleted all the timeshift snapshots, and it turns out that was the root cause! du and df finally match up and the problem is fixed, thanks so much! And yes I'll adjust timeshift's settings to be more space-efficient (probably involving only taking snapshots of important directories instead of the entire root directory).

Last edited by HumungusFungus (2023-03-21 14:33:51)

3beb6e7c46a615a · 2023-03-21 15:37:53

You're welcome.

I'd like to point out that if you take only selective snapshots you can no longer instantly restore your entire system to a past state by booting into a snapshot, which deprive snapshots of their value as backup tool, in my opinion. If restore is no longer instant anyway, you can turn to different backup strategies which provide more safety and reliability and do not consume local disk space (e.g. full offsite backups with borg, restic or similar tools).

Arch Linux

#1 2023-03-20 23:13:53

[SOLVED] Unable to update files with pacman, system thinks disk's full

#2 2023-03-20 23:25:53

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#3 2023-03-20 23:46:42

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#4 2023-03-21 07:40:54

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#5 2023-03-21 09:27:25

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#6 2023-03-21 10:05:25

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#7 2023-03-21 14:33:14

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

#8 2023-03-21 15:37:53

Re: [SOLVED] Unable to update files with pacman, system thinks disk's full

Board footer