Weird ghost writing to the disk (200gb+).

GhostVLK · 2024-11-01 23:35:01

Hi,

Yesterday, something started writing nonstop to my SD with the filesystem. I realized too late and i wasn't able to track which app was writing. Now, my fs drive is almost full, it added around 250 gb, but i can't manage to find which files and where were they written. If i check the size of /, it's around 116gb (the correct size), but if i check the dev where i have the fs, it shows around 375gb used.

My output for df -H and pretty much any other way shows / with 375g in use:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 480G 375G 103G 79% /

But the files on it are no bigger than 116gb, and that huge amount of written data appeared yesterday.

Is there any way to track what was written and where? (and to delete it somehow).

Trilby · 2024-11-01 23:40:52

Please give more concrete information: i.e., actual observations / commands and output not your interpretation.

For example, you say the files are bigger... what files? You say you don't know where this writing is happening, but if you are seeing big files, you'd know the file names - so that doesn't add up.

Broadly speaking, du / ncdu might be useful. You may also want / need to boot a live usb and examine the file system when its not in use.

V1del · 2024-11-01 23:42:11

use ncdu to find the actual files/spot that takes up space. Also note that if you use btrfs you might have some snapshots or so you could get rid of.

WorMzy · 2024-11-01 23:46:19

Mod note: not a kernel/hw issue, moving to NC.

GhostVLK · 2024-11-02 00:21:04

Trilby wrote:

Please give more concrete information: i.e., actual observations / commands and output not your interpretation.
For example, you say the files are bigger... what files? You say you don't know where this writing is happening, but if you are seeing big files, you'd know the file names - so that doesn't add up.
Broadly speaking, du / ncdu might be useful. You may also want / need to boot a live usb and examine the file system when its not in use.

I NEVER SAID i was seeing "bigger files", i literally described the issue and provided the data, i'm gonna say it again:

Up to yesterday, my used space in my filesystem drive (dev/sdb2) was around 116gb, yesterday, something happened (i don't know exactly what, it was 100% out of the blue, not related to any package upgrade or anything of the sort) but something started writing to that volume, at around.. 1gb per minute. It kept going on untill the used space (reported by df -H) reached 375gb, at that moment, i shut down the computer, so, whatever it was, stopped, but whatever was written, remains there. Today, i started looking for what was written, but i couldn't find those 260gb extra anywhere, the total size for the filesystem (checking with gdu, or any file manager) is shown as 116gb, but the used space in the volume is still shown as 375gb:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 480G 375G 103G 79% /

I'm checking now with ncdu to see if it shows me something different than gdu.

Trilby · 2024-11-02 00:29:42

Ah, well, you're right on one point: you didn't say there were bigger files. I misread the statement "But the files on it are no bigger than 116gb". But just the same, this is not a command or output, but your inference: where did you get this 116GB number from? And how do you know something was writing at 1GB per minute? How did you observe this.

These are questions I'd ask if I gave a damn about responding any further - but now I really don't. Even if I misread a small point, I was right about the need for concrete information, but you responded ... well, like this. That is an excellent way to get people to not want to help you.

Last edited by Trilby (2024-11-02 00:29:58)

GhostVLK · 2024-11-02 00:49:12

I wrote clearly what the issue was, you misunderstood, answered with a really obnoxious attitude, i clarified what you misunderstood, and you keep with the attitude. Now you are just tossing random questions, without really any argument.

About the second part.. uff, no pal.. i didn't just "close my eyes and guessed a number", i literally wrote that in my last topic, "I checked with gdu and with a couple file managers (nemo, nautilus, etc)". and now, with ncdu with the exact same result.

"How did i know it was writing at 1 gb per second?" Because when i realized that something was happening i started checking the dev size every minute or so and saw it was increasing at 1 gb per minute (more or less)!

I moved some files and deleted others to make more space, i reduced the total files in the volume to 88gb, but still df -H is reporting that there are 375gb in use in /dev/sdb2 that are not showing as files.

And as a PS, i never asked for YOUR help, i posted here in hopes that somebody could give me any insights on how to solve the issue, so, feel free to leave if you don't wanna contrib.

V1del wrote:

use ncdu to find the actual files/spot that takes up space. Also note that if you use btrfs you might have some snapshots or so you could get rid of.

Just checked with ncdu, it's the same as gdu or any other method, it's showing the total size of / as 116gb. I just deletod some files and moved others to make some space and i reduced the total file size to 88gb, but df -H is still showing the volume with 375gb in use.

Last edited by GhostVLK (2024-11-02 00:54:27)

skunktrader · 2024-11-02 01:03:24

GhostVLK wrote:

but something started writing to that volume, at around.. 1gb per minute. It kept going on untill the used space (reported by df -H) reached 375gb, at that moment, i shut down the computer, so, whatever it was, stopped, but whatever was written, remains there.

Run an fsck on the volume to see if the file system integrity is OK

seth · 2024-11-02 08:58:54

Regardless of the fsck, df/du disparity is typically because you've data in some mountpoint which gets hidden by the mounted partition.
We had a past case where the xsession or error log or something like that went wild but was opened before the user mounted their (encrypted?) $HOME so inspect your mountpoints (in doubt from some live distro, though if you only login as root on the console, you should™ be able to umount pretty much everything)

Then review your attitude, because

I NEVER SAID i was seeing "bigger files"

Yes, you did:

But the files on it are no bigger than 116gb

The intended meaning gets clear from your protest, but the literal statement is *not* that the files on the partition don't accumulate to more than 116gb

Therefore, as pointed our by Trilby, please don't paraphrase, https://bbs.archlinux.org/viewtopic.php?id=57855
This will work much better if you share the actual data (hence the sticky - in this case df -h , du -s and lsblk -f would likely be the most relevant if you don't find the compromising files in the mountpoint anyway) and on a formal note please also use [code][/code] tags, not "quote" tags. Edit your post in this regard.

Final note: if you find the bogus hidden file: don't just delete it - take a look what that is and its content might hint at the original problem.

GhostVLK · 2024-11-02 11:51:58

seth wrote:

Regardless of the fsck, df/du disparity is typically because you've data in some mountpoint which gets hidden by the mounted partition.
We had a past case where the xsession or error log or something like that went wild but was opened before the user mounted their (encrypted?) $HOME so inspect your mountpoints (in doubt from some live distro, though if you only login as root on the console, you should™ be able to umount pretty much everything)

Mmm, nope, not the case, i have mounted points in /mnt for other drives and a couple rclones mounts in my /home/user, but i unmounted everything before checking the total file size. Furthermore, the size in those mount points won't match the delta in this case (it's way more). Also, i've had the same mounted points for years. This was something specific.

seth wrote:

Then review your attitude, because
I NEVER SAID i was seeing "bigger files"
Yes, you did:
But the files on it are no bigger than 116gb
The intended meaning gets clear from your protest, but the literal statement is *not* that the files on the partition don't accumulate to more than 116gb

I never stated 'the files are getting bigger' as the other guy pointed out, i never stated anything, anywhere in my first post regarding that the size of the files has increased, in literally any way. I literally said "The files are no bigger than x and df is showing me the used space is y" (with y>x). Where in any of those statements it says "the files are getting bigger". Why it's so hard to admit that he misread?

For the output:

df -h:

Filesystem               Size  Used Avail Use% Mounted on
/dev/sdb2                447G  349G   96G  79% /
devtmpfs                 4.0M     0  4.0M   0% /dev
tmpfs                     24G  124M   24G   1% /dev/shm
efivarfs                 128K   45K   79K  36% /sys/firmware/efi/efivars
tmpfs                    9.4G  1.7M  9.4G   1% /run
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-udev-load-credentials.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-sysusers.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-vconsole-setup.service
/dev/sdb2                447G  349G   96G  79% /.snapshots
/dev/sdb2                447G  349G   96G  79% /home
tmpfs                     24G   73M   24G   1% /tmp
/dev/sdb2                447G  349G   96G  79% /var/cache/pacman/pkg
/dev/sdb2                447G  349G   96G  79% /var/log
/dev/sda5                222G  137G   86G  62% /mnt/C
/dev/sdd2                932G  708G  225G  76% /mnt/D
/dev/sdb1                510M  104M  407M  21% /boot
/dev/sdc2                954G  880G   75G  93% /mnt/E
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                    1.0M     0  1.0M   0% /run/credentials/systemd-sysctl.service
//192.168.1.108/public   7.3T  5.6T  1.7T  78% /mnt/Atlas_II
//192.168.1.109/public   3.6T  3.2T  436G  89% /mnt/Atlas_I
tmpfs                    1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs                    4.7G   96K  4.7G   1% /run/user/1000
oned:xxxx  5.0G  205M  4.8G   5% /home/ghost/OneDrive
dbox:xxxx        2.0G   81M  2.0G   4% /home/ghost/Dropbox

I replaced the rclone points with xxxx cause the names have work information.

lsblk -f

NAME   FSTYPE FSVER LABEL        UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ntfs         Recuperación 6EEE43EBEE43A9E3
├─sda2 ntfs         Recovery     C4DC0C09DC0BF488
├─sda3 vfat   FAT32              EA0E-5C42
├─sda4
├─sda5 ntfs        drvs         3AF8FE72F8FE2C2F                       85.6G    61% /mnt/C
└─sda6 ntfs                      A696B44496B416B1
sdb
├─sdb1 vfat   FAT32              7E7D-EE04                             406.3M    20% /boot
└─sdb2 btrfs                     c716879c-fd5d-4c69-88da-4273854b8d23   95.4G    78% /var/log
                                                                                     /var/cache/pacman/pkg
                                                                                     /home
                                                                                     /.snapshots
                                                                                     /
sdc
├─sdc1
└─sdc2 ntfs         drvr         4288EE9488EE85AF                       74.3G    92% /mnt/E
sdd
├─sdd1
└─sdd2 ntfs         drvt         D0BE2E9BBE2E79DC                      224.1G    76% /mnt/D
zram0                                                                                [SWAP]

du gets stuck, either with s or h flags, not sure if it's related.

seth · 2024-11-02 16:40:04

du gets stuck, either with s or h flags, not sure if it's related.

du will take a moment, that's normal.
You can however completely forget about those tools on btrfs, https://wiki.archlinux.org/title/Btrfs# … free_space

Edit: I really don't want to have that discussion, but your original assertion is ambigious at best. Schroedinger's cat is dead and alive, not nothing.
Yes, it's possible to guess what you likely meant, but guesswork on what data you're actually reporting isn't a very solid base for further action.

Last edited by seth (2024-11-02 16:44:04)

GhostVLK · 2024-11-02 18:35:09

Well, it's literally all i've got.. trust me, i've looked everywhere and i don't get the slightest clue of what could have been written or where, and/or how to get any more info about it . It still persists.

seth · 2024-11-02 20:41:17

Well, it's literally all i've got.. trust me, i've looked everywhere

Did you check the filesystem usage with the btrfs specific tools (what is absolutely necessary - reports of df and du are pretty much useless on btrfs)
What do they report?

WorMzy · 2024-11-02 22:11:42

Mod note: I've split off one of the less civil posts from this topic and opted to leave the topic open for now, but note that if the attitudes on display don't improve, then the whole thing is likely to end up in the dustbin.

https://terms.archlinux.org/docs/code-o … ther-users
https://terms.archlinux.org/docs/code-o … esponsible

GhostVLK · 2024-11-03 00:28:41

seth wrote:

Well, it's literally all i've got.. trust me, i've looked everywhere
Did you check the filesystem usage with the btrfs specific tools (what is absolutely necessary - reports of df and du are pretty much useless on btrfs)
What do they report?

Shows pretty much the same usage than df:

Overall:
    Device size:                 446.63GiB
    Device allocated:            368.63GiB
    Device unallocated:           78.00GiB
    Device missing:                  0.00B
    Device slack:                 16.00EiB
    Used:                        348.33GiB
    Free (estimated):             95.40GiB      (min: 56.40GiB)
    Free (statfs, df):            95.40GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              458.92MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:362.61GiB, Used:345.21GiB (95.20%)
Metadata,DUP: Size:3.00GiB, Used:1.56GiB (52.03%)
System,DUP: Size:8.00MiB, Used:64.00KiB (0.78%)

skunktrader wrote:

GhostVLK wrote:
but something started writing to that volume, at around.. 1gb per minute. It kept going on untill the used space (reported by df -H) reached 375gb, at that moment, i shut down the computer, so, whatever it was, stopped, but whatever was written, remains there.
Run an fsck on the volume to see if the file system integrity is OK

Did too, everything was ok, also checked for bad blocks, nothing.

Last edited by GhostVLK (2024-11-03 00:30:42)

seth · 2024-11-03 07:37:19

I don't think we've seen any acutal du data ever.
You asserted the results of ncdu and gdu but also claimed that du got "stuck"
So let's start with the obvious:

sudo du -hs /.snapshots

GhostVLK · 2024-11-03 10:24:25

Wait!! Everything up to /var has the right size, but du is now showing me 262gb in /var/lib which gdu/ncdu were NOT showing.

Let me see where is it located..

EDIT: Found it!! The crash.log on bitlbee is 261gb. Something must have happened that it started writing on it nonstop. However, nothing is showing me the file with that size, except du.

I'm gonna check if it's safe to just delete it.

Last edited by GhostVLK (2024-11-03 10:31:25)

GhostVLK · 2024-11-03 10:39:24

Well, i just deleted it, created a blank one, rebooted, and everything seems to be working fine, and the used space went back to normal.

Thanks a lot Seth!!!

seth · 2024-11-03 12:52:35

If you ran (only) du as root (sudo) it's likely down to an access rigths issue (ie. the other tools could not look into /var/lib/bitlbee and should™ have issued a warning about this)

GhostVLK · 2024-11-03 18:33:02

Well, yeah, that was my guess, but none of the other tools gave me any warning (du did, actually, that var required root access).

Arch Linux

#1 2024-11-01 23:35:01

Weird ghost writing to the disk (200gb+).

#2 2024-11-01 23:40:52

Re: Weird ghost writing to the disk (200gb+).

#3 2024-11-01 23:42:11

Re: Weird ghost writing to the disk (200gb+).

#4 2024-11-01 23:46:19

Re: Weird ghost writing to the disk (200gb+).

#5 2024-11-02 00:21:04

Re: Weird ghost writing to the disk (200gb+).

#6 2024-11-02 00:29:42

Re: Weird ghost writing to the disk (200gb+).

#7 2024-11-02 00:49:12

Re: Weird ghost writing to the disk (200gb+).

#8 2024-11-02 01:03:24

Re: Weird ghost writing to the disk (200gb+).

#9 2024-11-02 08:58:54

Re: Weird ghost writing to the disk (200gb+).

#10 2024-11-02 11:51:58

Re: Weird ghost writing to the disk (200gb+).

#11 2024-11-02 16:40:04

Re: Weird ghost writing to the disk (200gb+).

#12 2024-11-02 18:35:09

Re: Weird ghost writing to the disk (200gb+).

#13 2024-11-02 20:41:17

Re: Weird ghost writing to the disk (200gb+).

#14 2024-11-02 22:11:42

Re: Weird ghost writing to the disk (200gb+).

#15 2024-11-03 00:28:41

Re: Weird ghost writing to the disk (200gb+).

#16 2024-11-03 07:37:19

Re: Weird ghost writing to the disk (200gb+).

#17 2024-11-03 10:24:25

Re: Weird ghost writing to the disk (200gb+).

#18 2024-11-03 10:39:24

Re: Weird ghost writing to the disk (200gb+).

#19 2024-11-03 12:52:35

Re: Weird ghost writing to the disk (200gb+).

#20 2024-11-03 18:33:02

Re: Weird ghost writing to the disk (200gb+).

Board footer