[solved] What causes these changes in installed files?

sekret · 2023-01-05 23:26:17

Sorry for the bad thread title!

Here's what's going on. I want to watch my system very closely, so within the script I use for system updates

#!/bin/sh
pacman -Syu
pacman -Rcns $(pacman -Qdtq)
checkrebuild
auracle sync
pacman-ps
pwck
grpck
pacdiff
pacman -Qkk | grep ^error
journalctl -p 1 -xb
checkrebuild

I get this output, which is provided by "pacman -Qkk | grep ^error", which checks for changes in files that are controlled by pacman.

…
warning: btrfs-progs: /usr/lib/python3.10/site-packages/btrfsutil-6.1-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: btrfs-progs: /usr/lib/python3.10/site-packages/btrfsutil-6.1-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
…
warning: img2pdf: /usr/lib/python3.10/site-packages/img2pdf-0.4.4-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: img2pdf: /usr/lib/python3.10/site-packages/img2pdf-0.4.4-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: img2pdf: /usr/lib/python3.10/site-packages/img2pdf-0.4.4-py3.10.egg-info/zip-safe (MD5 checksum mismatch)
warning: img2pdf: /usr/lib/python3.10/site-packages/img2pdf-0.4.4-py3.10.egg-info/zip-safe (SHA256 checksum mismatch)
…
warning: pybind11: /usr/lib/python3.10/site-packages/pybind11-2.10.3-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: pybind11: /usr/lib/python3.10/site-packages/pybind11-2.10.3-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: pybind11: /usr/lib/python3.10/site-packages/pybind11-2.10.3-py3.10.egg-info/not-zip-safe (MD5 checksum mismatch)
warning: pybind11: /usr/lib/python3.10/site-packages/pybind11-2.10.3-py3.10.egg-info/not-zip-safe (SHA256 checksum mismatch)
warning: python-flatbuffers: /usr/lib/python3.10/site-packages/flatbuffers-23.1.4-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: python-flatbuffers: /usr/lib/python3.10/site-packages/flatbuffers-23.1.4-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: python-google-auth: /usr/lib/python3.10/site-packages/google_auth-2.15.0-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: python-google-auth: /usr/lib/python3.10/site-packages/google_auth-2.15.0-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: python-matplotlib: /usr/lib/python3.10/site-packages/matplotlib-3.6.2-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: python-matplotlib: /usr/lib/python3.10/site-packages/matplotlib-3.6.2-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: python-pooch: /usr/lib/python3.10/site-packages/pooch-1.6.0.dist-info/zip-safe (MD5 checksum mismatch)
warning: python-pooch: /usr/lib/python3.10/site-packages/pooch-1.6.0.dist-info/zip-safe (SHA256 checksum mismatch)
warning: python-pygments: /usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/dependency_links.txt (MD5 checksum mismatch)
warning: python-pygments: /usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/dependency_links.txt (SHA256 checksum mismatch)
warning: python-pygments: /usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/not-zip-safe (MD5 checksum mismatch)
warning: python-pygments: /usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/not-zip-safe (SHA256 checksum mismatch)
…

I checked the packages for install scripts, which could cause this. None of them have any install scripts (edit: except btrfs-progs does!).
I've removed those files (edit: to be absolutely sure they are written again) and reinstalled the packages. Still, those warnings appear.

Then I looked into them and into the original file in an unpacked pacman package. They are all just empty. And very strange, manually comparing the checksums … they are the same.
Here's an example for python-google-auth:

$ md5sum dependency_links.txt
68b329da9893e34099c7d8ad5cb9c940  dependency_links.txt
$ md5sum /usr/lib/python3.10/site-packages/google_auth-2.15.0-py3.10.egg-info/dependency_links.txt
68b329da9893e34099c7d8ad5cb9c940  /usr/lib/python3.10/site-packages/google_auth-2.15.0-py3.10.egg-info/dependency_links.txt

Ok strange. AFAIK pacman only saves files to the filesystem and install scripts could change them afterwards. But there's no install scripts here.
Since I have a LOT more packages which have files inside /usr/lib/python3.10/site-packages which don't produce this warning I guess this started at a certain point when those packages received an update.

So I wanted to know if a simple reinstallation would start this (making a pacman update the possible culprit). So I reinstalled python-anyio as an example and checked if this warning appears. No, it doesn't.
Next option: I rebuilt python-anyio (in a clean chroot), installed it and checked the output. The warning doesn't appear either.

I don't understand this warning. And I don't know where the problem is located exactly. Unfortunately I also don't know when this started to appear exactly. It might be no big issue, but it's to strange to keep it to myself and because I absolutely don't get it I haven't filed it as a bug report yet.

Last edited by sekret (2023-03-07 20:49:05)

Trilby · 2023-01-05 23:39:45

First, I can confirm the same results. There are countless benign reasons for such a difference, do you need to identify specifically which one accounts for this difference? Note that these are warnings, not errors. Then take that bit of information and think about the contrast between what you intended your script to do and what it is actually doing and either get rid of the pipe to grep or fix it so it actually does what you intended.

Last edited by Trilby (2023-01-05 23:41:39)

sekret · 2023-01-06 12:00:56

Ok, you're right about the command, it's really bad, so thanks for the hint!

I know those are just warnings, so there's nothing broken here. But imho it's bad practice to just ignore warnings. I see them as a hint to a potential issue which might have the potential to break things eventually. But I'm no computer scientist, so my point is more of a philosophical one.

cfr · 2023-01-06 23:01:52

Trilby wrote:

There are countless benign reasons for such a difference, do you need to identify specifically which one accounts for this difference?

I don't need to know, but I would like to know. I certainly don't suspect anything non-benign - if something malicious was responsible, it would be doing much worse than this. But I am really curious about it.

It makes me curious, in particular, about how pacman calculates the checksums for comparison. Apparently extracting the files and calculating them is not equivalent to whatever pacman does, which isn't especially disturbing, but it is odd that the results aren't equivalent. At least, it seems odd to me, being ignorant of the process.

Trilby · 2023-01-06 23:37:06

I just tested with python-pygments. I confirmed that installing it from the repos results in `pacman -Qkk python-pygments` emitting this warning. I also confirmed that the md5sum listed in the local database mtree file was indeed different from that of the dependency_list.txt file.

Then I grabbed python-pygments from the ABS and built it myself. I then compared the md5sum for that file in the new packages mtree file and found it to be the same as the md5sum of the installed file. And when this newly built package was installed it "fixed" the md5sum and other checksum values in the local database mtree file and `pacman -Qkk` no longer gave these warnings.

I then re-installed python-pygments from the repos and confirmed the warnings returned.

It would seem that there is indeed an issue of some sort with how these repo packages were built as the files did not change after installation, but the packages' MTREE files have incorrect checksum values. While irrelevant for these particular files, it could be concerning if the same error could hit other files.

Or more concisely, a demonstration of an error in a repo package:

# get the python-pygments package from the repos, extract it:
$ bsdtar xf python-pygments-2.14.0-1-any.pkg.tar.zst

# checksum an actual file:
$ md5sum usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/dependency_links.txt 
68b329da9893e34099c7d8ad5cb9c940  usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/dependency_links.txt

# confirm that this differs from the recorded checksum in .MTREE:
$ zgrep dependency_links.txt .MTREE 
./usr/lib/python3.10/site-packages/Pygments-2.14.0-py3.10.egg-info/dependency_links.txt time=1672687238.0 size=1 md5digest=93b885adfe0da089cdf634904fd59f71 sha256digest=6e340b9cffb37a989ca544e6bb780a2c78901d3fb33738768511a30617afa01d

So while many benign processes can and will change packaged files after installation, a file in a package not matching the packages MTREE prior to installation is more concerning and seems indicative of a bug. The exception may be if this can only happen with empty files ... and I vaguely remember hearing something recently about a change in how empty files are handled for checksums though I could be imagining it; I'll have to look for that. If this mismatch between a package's content and mtree data could be found for a non-empty file, then I'd definitely be concerned.

EDIT: FWIW, the checksum of the file is the checksum of a single newline character, while the recorded value in the MTREE file is the checksum of a single null-byte. Ironically, neither of them are the checksum of an actual empty file!

Last edited by Trilby (2023-01-06 23:54:50)

cfr · 2023-01-07 05:26:53

Trilby wrote:

EDIT: FWIW, the checksum of the file is the checksum of a single newline character, while the recorded value in the MTREE file is the checksum of a single null-byte. Ironically, neither of them are the checksum of an actual empty file!

Thanks for looking into this. The example I looked at involved files with a single newline i.e. not literally empty and 'file' does not report them empty.

sekret · 2023-01-07 14:44:55

I didn't know about those MTREE files, so that's why I wasn't able to dig into this deeper. Thanks Trilby for your deeper investigation and analysis.

sekret · 2023-03-07 20:48:51

Ok, looks like this has been fixed, I don't get those messages anymore.

Arch Linux

#1 2023-01-05 23:26:17

[solved] What causes these changes in installed files?

#2 2023-01-05 23:39:45

Re: [solved] What causes these changes in installed files?

#3 2023-01-06 12:00:56

Re: [solved] What causes these changes in installed files?

#4 2023-01-06 23:01:52

Re: [solved] What causes these changes in installed files?

#5 2023-01-06 23:37:06

Re: [solved] What causes these changes in installed files?

#6 2023-01-07 05:26:53

Re: [solved] What causes these changes in installed files?

#7 2023-01-07 14:44:55

Re: [solved] What causes these changes in installed files?

#8 2023-03-07 20:48:51

Re: [solved] What causes these changes in installed files?

Board footer