pacman writes empty files on update

H3g3m0n · 2023-12-19 23:26:32

I have had this problem for YEARS now. I would like to find and fix the underlying cause. Maybe this should be a bug report for pacman but I thought I would check here first.

Randomly a routine "pacman -Syu" update will replace files (notably library files) with empty ones (on rare occasions I have seen truncated too but it might be a separate issue). ldconfig will complain about the empty files if I check the pacman output (but often I miss it since I will fire-and-forget the upgrade in another desktop tab).

Normally this results in pages of ldconfig errors being thrown by pacman like the following:

ldconfig: File /usr/lib/libnspr4.so is empty, not checked.

It seems to be large batches of libraries from a single update.

Often the empty files will result in a system that is unbootable or unable to connect to the network/wifi which leads to the pain of pulling out a bootable usb, manually chrooting into it then reinstalling the effected packages as shown with "pacman -Qkk" (slow and with some false positives), or the error output of ldconfig. (Since the packages are on a shared NFS I can't just reinstall packages from the local cache). Although I guess I could also end up in a situation where I don't notice the empty files and some program just doesn't work.

This has been happening on 2 different systems so I don't believe it's hardware related.

I'm not sure what causes this. The relevant commonality between the systems are they are both using btrfs. And both are mounting /var/cache/pacman/pkg from a NFS mount (don't see how this would change the files written since it should only verify then extracted files from here). The only other thing I could think of would be low space (shouldn't pacman handle this in some way?) or something not syncing before reboot.

I don't seem to be the only one with the issue. (for example). But the discussions are around just fixing the system (which isn't a problem for me since I have done it often enough) rather than fixing the underlying cause. It's weird that it only seems to be library files.

Is there anyway to force pacman to verify files after it has written them? What about force a 'sync' after running?

Scimmia · 2023-12-19 23:46:36

This happens when there's an issue with the filesystem, usually from a crash or power loss, but it could be from hardware problems or fs bugs or other reasons as well. It's not a pacman problem.

Verifying the files before reboot doens't help, as they're in the disk cache and won't show a problem. You can use a libalpm hook for force a sync after the transaction if you'd like.

H3g3m0n · 2023-12-20 00:35:57

Scimmia wrote:

This happens when there's an issue with the filesystem, usually from a crash or power loss, but it could be from hardware problems or fs bugs or other reasons as well. It's not a pacman problem.
Verifying the files before reboot doens't help, as they're in the disk cache and won't show a problem. You can use a libalpm hook for force a sync after the transaction if you'd like.

Thanks.

Fairly sure it's not from crashing, both systems are stable.

Also don't generally see powerloss (the laptop might on occasion if the battery dies, but it's normally plugged in 24/7 and it happening right when updating or shortly after is unlikely).

There is some possibility that I might accidentally turn of the system during the upgrade but I would think that would result in pacman being in a half configured state requiring manual intervention and forcing me to resume the upgrade? But that should still sync the files since it would go through the shutdown process.

Even that would be rare since the upgrades are normally quick to apply. Unless it's possible for files to sit around without being sync'd in the background for over half an hour?

My best guess is it's some btrfs setting. I previously had issues with the defrag option. I'm currently running with

relatime,compress=lzo,ssd,space_cache,discard

I have found https://github.com/andrewgregory/pachoo … /sync.hook so hopefully that fixes it.

Last edited by H3g3m0n (2023-12-20 00:40:56)

seth · 2023-12-20 09:35:26

Why are you not using the default "discard=async"?
( Though caveat on even that: https://wiki.archlinux.org/title/Btrfs# … e_activity )

The wiki wrote:

Continuous TRIM is not the most preferred way to issue TRIM commands among the Linux community

A "sync" hook is maybe a crutch for that one problem, but if you're (likely) systematically loosing data (seen the big fat red warning in https://wiki.archlinux.org/title/Solid_ … nuous_TRIM

queued TRIM command execution was blacklisted due to serious data corruption

If your drive is affected but not in the blacklist, you're systematically shredding your data.
https://en.wikipedia.org/wiki/Trim_(com … advantages

I'd lose that first.

Then generally check the integrity of the FS (btrfs isn't exactly the most robust FS…) and https://wiki.archlinux.org/title/SMART (though the latter being less likely the cause if multiple systems are affected)

Last edited by seth (2023-12-20 09:35:39)

Nebulosa · 2023-12-20 21:59:44

H3g3m0n wrote:

relatime,compress=lzo,ssd,space_cache,discard

As said there: https://btrfs.readthedocs.io/en/latest/ … ation.html

relative - default option, not need to explicit set

ssd - default: SSD autodetected, better to change ssd_spread

space_cache - also better delete (for default v2), or set space_cache=v2

discard - again, better delete for default async value.

So what really need to set is just 'compress=lzo,ssd_spread' all rest will be default options that gives you max performance.

Arch Linux

#1 2023-12-19 23:26:32

pacman writes empty files on update

#2 2023-12-19 23:46:36

Re: pacman writes empty files on update

#3 2023-12-20 00:35:57

Re: pacman writes empty files on update

#4 2023-12-20 09:35:26

Re: pacman writes empty files on update

#5 2023-12-20 21:59:44

Re: pacman writes empty files on update

Board footer