[SOLVED] SSD filling up because of backup failure

PopeRigby · 2020-04-24 00:25:35

I setup a backup system a few weeks ago, that backs up part of my system to Google Drive once a day using duplicity:

#!/bin/sh

export PASSPHRASE="PASSPHRASE"

# runs daily and does an incremental backup
/usr/bin/duplicity --encrypt-key ENCRYPT_KEY --include-filelist /usr/local/lib/globbing_list / rclone://gdrive:/backup

globbing_list:

- /bin
- /boot
- /dev
- /Desktop
/etc
- /etc/.pwd.lock
- /home
/home/username/.config
/home/username/.local/bin
/home/username/.ssh
/home/username/.var
/home/username/.mozilla
- /lib
- /lib64
- /lost+found
- /media
- /mnt
- /opt
- /proc
/root
- /run
- /sbin
- /srv
- /sys
- /tmp
- /usr
/usr/local
/var/lib
- /var

It's been working just fine, until now. Yesterday afternoon, I noticed that my system was slowing to a crawl, so for some reason that I don't remember, I decided to check the status of the systemd service that runs my backup. This was the output:

● backup-inc.service - Run incremental duplicity backup
     Loaded: loaded (/etc/systemd/system/backup-inc.service; static; vendor preset: disabled)
     Active: active (running) since Thu 2020-04-23 00:00:26 PDT; 16h ago
TriggeredBy: ● backup-inc.timer
   Main PID: 16082 (encrypt_backup_)
      Tasks: 4 (limit: 19138)
     Memory: 6.1G
     CGroup: /system.slice/backup-inc.service
             ├─16082 /bin/sh /usr/local/bin/encrypt_backup_incremental
             ├─16086 python3 /usr/bin/duplicity --encrypt-key ENCRYPT_KEY --include-filelist /usr/local/lib/globbing_list / rclone://gdrive:/hostname_backup
             └─52173 gpg --passphrase-fd 22 --logger-fd 16 --batch --no-tty --recipient ENCRYPT_KEY --no-secmem-warning --ignore-mdc-error --pinentry-mode=loopback --encrypt

Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/prev
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/prev
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/smack/current
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/smack/current
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/sockcreate
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/sockcreate
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/clear_refs
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/clear_refs
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Input/output error getting delta for /var/lib/dhcpcd/proc/1246/task/1246/mem
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Input/output error getting delta for /var/lib/dhcpcd/proc/1246/task/1246/mem

I restarted to see if it would fix it, and then forgot about it. Today, I realized that my SSD only had 300 GB out of 1 TB available. The day before that, it had 500 GB available, and I didn't install anything, so I was very confused. I checked KDE Filelight, and as you can see, my root directory is only taking up 280.2 GB:
https://i.imgur.com/cHUweWE.png

(I don't know what's taking up the other 500 GB, but that's something I can figure out later. What matters is the 200 GB that just went missing)

My theory is that it's failing to connect to Google Drive for some reason, and then storing the backup on my SSD. But I don't know why it wouldn't show up in filelight.

Edit: Another weird thing is, it seems to be backing up alright. Recently uploaded files are showing up in my Google Drive.

Mod Edit - Replaced oversized image with link.
CoC - Pasting pictures and code

Last edited by PopeRigby (2020-04-26 22:53:30)

seth · 2020-04-24 06:41:02

Please replace the oversized image w/ a thumbnail or link and post the outputs of "df -h", "df -hi" and "sudo du -hs /*"
A common trap is that you've shadowed files by mounting into their directory, ie.

dd if=/dev/zero of=/tmp/foo/bar count=1M
du -hs /tmp/foo
mount /dev/sdXn /tmp/foo
du -hs /tmp/foo

PopeRigby · 2020-04-24 17:07:15

seth wrote:

Please replace the oversized image w/ a thumbnail or link

Sorry about that. Fixed.

and post the outputs of "df -h", "df -hi" and "sudo du -hs /*"

df -h:

Filesystem      Size  Used Avail Use% Mounted on
dev             7.8G     0  7.8G   0% /dev
run             7.8G  1.5M  7.8G   1% /run
/dev/sdc2       900G  520G  335G  61% /
tmpfs           7.8G  149M  7.7G   2% /dev/shm
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           7.8G  117M  7.7G   2% /tmp
/dev/sdc1       511M   96M  416M  19% /boot
tmpfs           1.6G   56K  1.6G   1% /run/user/1000

df -hi:

Filesystem     Inodes IUsed IFree IUse% Mounted on
dev              2.0M   530  2.0M    1% /dev
run              2.0M   894  2.0M    1% /run
/dev/sdc2         58M  1.1M   57M    2% /
tmpfs            2.0M   167  2.0M    1% /dev/shm
tmpfs            2.0M    18  2.0M    1% /sys/fs/cgroup
tmpfs            2.0M    68  2.0M    1% /tmp
/dev/sdc1           0     0     0     - /boot
tmpfs            2.0M    92  2.0M    1% /run/user/1000

sudo du -hs /*:

0       /bin
96M     /boot
4.0K    /Desktop
4.1M    /dev
12M     /etc
282G    /home
0       /lib
0       /lib64
16K     /lost+found
4.0K    /media
12K     /mnt
370M    /opt
du: cannot read directory '/proc/1422/task/1422/net': Invalid argument
du: cannot read directory '/proc/1422/net': Invalid argument
du: cannot access '/proc/79355/task/79355/fd/4': No such file or directory
du: cannot access '/proc/79355/task/79355/fdinfo/4': No such file or directory
du: cannot access '/proc/79355/fd/3': No such file or directory
du: cannot access '/proc/79355/fdinfo/3': No such file or directory
0       /proc
205G    /root
du: cannot access '/run/user/1000/doc': Permission denied
1.5M    /run
0       /sbin
12K     /srv
0       /sys
117M    /tmp
16G     /usr
19G     /var

Slithery · 2020-04-24 17:30:18

PopeRigby wrote:

seth wrote:
Please replace the oversized image w/ a thumbnail or link
Sorry about that. Fixed.

No, you didn't. Have fixed it for you...

seth · 2020-04-24 20:10:23

/dev/sdc2       900G  520G  335G  61% /

The only significant load is on your root partition (520GB out of 900GB) lost in

282G    /home
…
205G    /root

(/usr is the OS, reasonable size; /var most likely blown up by the pacman cache, ie. packages from various updates that you can clean up "pacman -Sc")

There's no inode related issue, so the question is what blows up /home and /root - notably /root should™ rather not take so much space.

sudo du -hs /root/*

and then navigate down the biggest paths.

PopeRigby · 2020-04-24 20:24:53

Got it. On another note, my free space jumped up to 568 GB for reasons unknown, but that still doesn't explain where 156 GB has gone. Maybe I'm getting confused. Does the size of /home not get counted when calculating the size of of the root directory?

Anyway. here's the output of "sudo du -hs /root/*"

4.0K    /root/Desktop
4.0K    /root/Documents
4.0K    /root/Downloads
4.0K    /root/Music
4.0K    /root/Pictures
4.0K    /root/Public
4.0K    /root/Templates
4.0K    /root/Videos

Probably none of those then. But, when I just do "sudo du /root", it seems that the thing taking up the most space is the duplicity cache.

19G     root/.cache/duplicity

That goes along with my theory that it's just storing the backup on my computer because it can't push it to Google Drive, most likely because of:

Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/prev

Last edited by PopeRigby (2020-04-24 20:25:15)

seth · 2020-04-24 20:29:32

Ah. It's in some hidden directory.

du -hs /root/.*

Edit: is your hostname "hostname"?

Last edited by seth (2020-04-24 20:30:17)

PopeRigby · 2020-04-24 20:30:50

That gives me:

8.0K    /root/.bash_history
4.0K    /root/.bash_profile
19G     /root/.cache
120K    /root/.config
8.0K    /root/.emacs.d
4.0K    /root/.gist.bkp
4.0K    /root/.gitconfig
4.0K    /root/.git-credentials
52K     /root/.gnupg
4.0K    /root/.histfile
88K     /root/.local
4.0K    /root/.nvidia-settings-rc
8.0K    /root/.ssh
8.0K    /root/.vim
12K     /root/.viminfo
4.0K    /root/.wget-hsts
0       /root/.Xauthority
48K     /root/.zcompdump
4.0K    /root/.zshrc

Slithery · 2020-04-24 20:47:16

seth wrote:

is your hostname "hostname"?

seth · 2020-04-24 21:18:04

We also have ~185GB unaccounted for in the /root directory (btw. "/root" != "/", the latter being the "root" directory. Sorry I missed your former question)
Is this a btrfs partition?

PopeRigby · 2020-04-24 21:24:16

Slithery wrote:

seth wrote:
is your hostname "hostname"?

No. I just didn't want to put my real hostname.

seth wrote:

We also have ~185GB unaccounted for in the /root directory (btw. "/root" != "/", the latter being the "root" directory. Sorry I missed your former question)
Is this a btrfs partition?

No, just ext4.

seth · 2020-04-25 07:44:19

https://wiki.archlinux.org/index.php/So … drive#TRIM ?
Look around in /root, though

my free space jumped up to 568 GB for reasons unknown, but that still doesn't explain where 156 GB has gone.

Updated

df -h
sudo du -hs /*

?

PopeRigby · 2020-04-25 18:50:54

seth wrote:

Look around in /root, though

/root takes up 19 GB of space, and almost all of the space is being taken up by /root/.cache/duplicity

df -h

Filesystem      Size  Used Avail Use% Mounted on
dev             7.8G     0  7.8G   0% /dev
run             7.8G  1.5M  7.8G   1% /run
/dev/sdc2       900G  294G  561G  35% /
tmpfs           7.8G  255M  7.6G   4% /dev/shm
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           7.8G  1.6M  7.8G   1% /tmp
/dev/sdc1       511M   96M  416M  19% /boot
tmpfs           1.6G   68K  1.6G   1% /run/user/1000

sudo du -hs /*

0       /bin
96M     /boot
4.0K    /Desktop
6.1M    /dev
12M     /etc
242G    /home
0       /lib
0       /lib64
16K     /lost+found
4.0K    /media
12K     /mnt
370M    /opt
du: cannot read directory '/proc/1492/task/1492/net': Invalid argument
du: cannot read directory '/proc/1492/net': Invalid argument
du: cannot access '/proc/47286/task/47286/fd/4': No such file or directory
du: cannot access '/proc/47286/task/47286/fdinfo/4': No such file or directory
du: cannot access '/proc/47286/fd/3': No such file or directory
du: cannot access '/proc/47286/fdinfo/3': No such file or directory
0       /proc
19G     /root
du: cannot access '/run/user/1000/doc': Permission denied
1.5M    /run
0       /sbin
12K     /srv
0       /sys
1.6M    /tmp
16G     /usr
19G     /var

GNOME disks is reporting that my SSD has 651 GB of 983 GB of available on my / partition, but again, my file manager isn't reflecting that.

seth · 2020-04-25 19:01:43

my file manager isn't reflecting that.

What file manager reflects what instead and how?

Gnome disks should™ btw. say 561GB (at least that's what df says.)
The 900-294-561 gap is because the fs reserves some space for exclusive root access and itself, 900 ./. 983GB is likely just JEDEC ./. SI, see https://en.wikipedia.org/wiki/Mebibyte

PopeRigby · 2020-04-25 19:07:38

seth wrote:

my file manager isn't reflecting that.
What file manager reflects what instead and how?

I'm using Dolphin. Dolphin says I have 560.5 GB available:
https://i.imgur.com/6V095Do.png

Gnome disks should™ btw. say 561GB (at least that's what df says.)

GNOME disks says I have 651 GB available, and I have no idea why:
https://i.imgur.com/KzqaRg3.png

The 900-294-561 gap is because the fs reserves some space for exclusive root access and itself, 900 ./. 983GB is likely just JEDEC ./. SI, see https://en.wikipedia.org/wiki/Mebibyte

Yeah. The missing 17 GB is for my swap partition. Nothing to worry about there.

seth · 2020-04-25 19:14:27

Dolphin shows the values in IEC (GiB, same value as JEDEC. A magnitude is 2^10, not 10^3) gparted in SI (10^3)
So 561GiB are 602GB
Since the you're actually just using 294GiB of 900GiB there 606GiB free which is about 650GB

606*(1024^3)/(10^9)

Edit: available != free, as mentioned you don't get access to all inodes because the filesystem keeps some for maintainance.

Last edited by seth (2020-04-25 19:16:23)

PopeRigby · 2020-04-25 19:15:44

seth wrote:

Dolphin shows the values in IEC (GiB, same value as JEDEC. A magnitude is 2^10, not 10^3) gparted in SI (10^3)
So 561GiB are 602GB
Since the you're actually just using 294GiB of 900GiB there 606GiB free which is about 650GB
606*(1024^3)/(10^9)

I have no idea what you just said but I trust you know what you're talking about. So is that all my disk space accounted for?

seth · 2020-04-25 19:20:50

Yes.

Disk space used to be measured in units of 2^n (8 bit form one byte, 1024 bytes one kilobyte)
Obviously that's not an overly correct usage of the units (1 kilogramm is 1000 gramm, not 1024 gramm) and because it allows to tout bigger numbers, disk vendors at some point switched to talking in SI units, while all OS were still talking in JEDEC.
The IEC then allowed to keep the JEDEC units but introduced new prefixes (which is why 1GiB != 1GB)

Feel free to google the topic - it's a mess

PopeRigby · 2020-04-25 19:23:57

seth wrote:

Yes.
Disk space used to be measured in units of 2^n (8 bit form one byte, 1024 bytes one kilobyte)
Obviously that's not an overly correct usage of the units (1 kilogramm is 1000 gramm, not 1024 gramm) and because it allows to tout bigger numbers, disk vendors at some point switched to talking in SI units, while all OS were still talking in JEDEC.
The IEC then allowed to keep the JEDEC units but introduced new prefixes (which is why 1GiB != 1GB)
Feel free to google the topic - it's a mess

Sounds like a mess. So now that the disk usage is sorted, do you have any idea how to fix the:

Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/prev
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/prev
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/smack/current
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/smack/current
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/sockcreate
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/attr/sockcreate
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/clear_refs
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Invalid argument getting delta for /var/lib/dhcpcd/proc/1246/task/1246/clear_refs
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Input/output error getting delta for /var/lib/dhcpcd/proc/1246/task/1246/mem
Apr 23 16:50:28 hostname encrypt_backup_incremental[16086]: Error Input/output error getting delta for /var/lib/dhcpcd/proc/1246/task/1246/mem

seth · 2020-04-25 19:30:59

Most likely for the recent dhcpcd 9.x update
Since this seems to symlink or bind something from the /proc FS you can just filter it out.

loqs · 2020-04-25 19:32:57

https://bbs.archlinux.org/viewtopic.php … 8#p1900838

PopeRigby · 2020-04-25 19:33:17

seth wrote:

Most likely for the recent dhcpcd 9.x update
Since this seems to symlink or bind something from the /proc FS you can just filter it out.

Will filtering it out help the slowdown and dumping tons of files into /root/.cache/duplicity?

PopeRigby · 2020-04-25 19:33:56

loqs wrote:

https://bbs.archlinux.org/viewtopic.php … 8#p1900838

Oh, thanks! I'll check that out.

PopeRigby · 2020-04-25 20:16:57

I followed through the thread to fix it, but I can't exclude /var/lib/dhcpcd because it keeps giving me this error:

Reading globbing filelist /usr/local/lib/globbing_list
Last selection expression:
    Command-line include glob: /var/lib/upower
only specifies that files be included.  Because the default is to
include all files, the expression is redundant.  Exiting because this
probably isn't what you meant.

This is my include list:

- /bin
- /boot
- /dev
- /Desktop
/etc
- /etc/.pwd.lock
- /home
/home/user/.config
/home/user/.local/bin
/home/user/.ssh
/home/user/.var
/home/user/.mozilla
- /lib
- /lib64
- /lost+found
- /media
- /mnt
- /opt
- /proc
/root
- /root/.cache
- /run
- /sbin
- /srv
- /sys
- /tmp
- /usr
/usr/local
- /var
/var/lib/AccountsService
/var/lib/alsa
/var/lib/arpd
/var/lib/boltd
/var/lib/colord
/var/lib/dbus
/var/lib/geoclue
/var/lib/krb5kdc
/var/lib/machines
/var/lib/misc
/var/lib/mysql
/var/lib/NetworkManager
/var/lib/os-prober
/var/lib/PackageKit
/var/lib/pacman
/var/lib/portables
/var/lib/private
/var/lib/samba
/var/lib/sddm
/var/lib/systemd
/var/lib/texmf
/var/lib/udisks2
/var/lib/upower

It should be working (it was working for my home folder), because I'm saying to exclude /var, but then override that to include a few files from /var/lib.

seth · 2020-04-25 20:29:13

idk how this works but you previously had

/var/lib
- /var

and apparently /var/lib was picked up, so maybe try trailing "- /var" ?

Arch Linux

#1 2020-04-24 00:25:35

[SOLVED] SSD filling up because of backup failure

#2 2020-04-24 06:41:02

Re: [SOLVED] SSD filling up because of backup failure

#3 2020-04-24 17:07:15

Re: [SOLVED] SSD filling up because of backup failure

#4 2020-04-24 17:30:18

Re: [SOLVED] SSD filling up because of backup failure

#5 2020-04-24 20:10:23

Re: [SOLVED] SSD filling up because of backup failure

#6 2020-04-24 20:24:53

Re: [SOLVED] SSD filling up because of backup failure

#7 2020-04-24 20:29:32

Re: [SOLVED] SSD filling up because of backup failure

#8 2020-04-24 20:30:50

Re: [SOLVED] SSD filling up because of backup failure

#9 2020-04-24 20:47:16

Re: [SOLVED] SSD filling up because of backup failure

#10 2020-04-24 21:18:04

Re: [SOLVED] SSD filling up because of backup failure

#11 2020-04-24 21:24:16

Re: [SOLVED] SSD filling up because of backup failure

#12 2020-04-25 07:44:19

Re: [SOLVED] SSD filling up because of backup failure

#13 2020-04-25 18:50:54

Re: [SOLVED] SSD filling up because of backup failure

#14 2020-04-25 19:01:43

Re: [SOLVED] SSD filling up because of backup failure

#15 2020-04-25 19:07:38

Re: [SOLVED] SSD filling up because of backup failure

#16 2020-04-25 19:14:27

Re: [SOLVED] SSD filling up because of backup failure

#17 2020-04-25 19:15:44

Re: [SOLVED] SSD filling up because of backup failure

#18 2020-04-25 19:20:50

Re: [SOLVED] SSD filling up because of backup failure

#19 2020-04-25 19:23:57

Re: [SOLVED] SSD filling up because of backup failure

#20 2020-04-25 19:30:59

Re: [SOLVED] SSD filling up because of backup failure

#21 2020-04-25 19:32:57

Re: [SOLVED] SSD filling up because of backup failure

#22 2020-04-25 19:33:17

Re: [SOLVED] SSD filling up because of backup failure

#23 2020-04-25 19:33:56

Re: [SOLVED] SSD filling up because of backup failure

#24 2020-04-25 20:16:57

Re: [SOLVED] SSD filling up because of backup failure

#25 2020-04-25 20:29:13

Re: [SOLVED] SSD filling up because of backup failure

Board footer