[SOLVED] zstd decompression failed: Restored data doesn't match checks

thewisenoob · 2023-11-05 21:54:25

error: could not extract /usr/lib/libumfpack.so.6.2.2 (Zstd decompression failed: Restored data doesn't match checksum)
error: problem occurred while upgrading suitesparse
error: could not commit transaction
error: failed to commit transaction (transaction aborted)
Errors occurred, no packages were upgraded.

When I update my system I get this error on random packages. Last time I updated a couple weeks ago this did not happen. It also breaks my system because vmlinuz and initramfs are not no longer on the boot partition because of pacman erroring partway through the update. I can boot again if I reinstall the linux package from the Arch installer and then revert my system via Timeshift, but the error happens every time I try to update and the package it fails on is inconsistent.

I checked my nvme for bad sectors and ran memtest86+, but everything came back clean.

Anybody have any idea what is going on? Is there any info I could add to this post to help?

Thanks <3

Last edited by thewisenoob (2023-11-07 18:41:57)

seth · 2023-11-06 14:24:18

Are you running out of disk or RAM?
Can you decompress the package manually?

thewisenoob · 2023-11-06 14:33:32

seth wrote:

Are you running out of disk or RAM?
Can you decompress the package manually?

My 4TB drive has almost 3TB of free space. The only partition that is not my main one is the boot partition and it's nowhere near full either. I've got 64GB of RAM, so it's definitely not that either. This error happens if I chroot into my install as well.

I'll give manually installing the package a try later today, but since it happens on random packages and sometimes the one it fails on works on another try, I'm not sure that's going to help. I've tried deleting all of the cached packages too, but redownloading them doesn't fix this issue for me.

seth · 2023-11-06 15:15:07

I didn't mean to manually install the package.
I meant to try to decompress the file in the cache that caused the issue with "zstd -d" to see whether this is only a problem in the context of pacman decompression.
If yes, try "--single-thread", though afaiu that only applies to compression.

Assuming it's not the nvme itself, what filesystem do you use? mount parameters?

thewisenoob · 2023-11-06 15:28:39

seth wrote:

I didn't mean to manually install the package.
I meant to try to decompress the file in the cache that caused the issue with "zstd -d" to see whether this is only a problem in the context of pacman decompression.
If yes, try "--single-thread", though afaiu that only applies to compression.
Assuming it's not the nvme itself, what filesystem do you use? mount parameters?

Decompressing manually did not cause the error I've been seeing. I tried it with several packages several times that I remembered the update had previously given that zstd error on.

My main partition is ext4. I'm not 100% sure what you mean by mount parameters. Do you mean what is in my fstab? Here it is:

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p2 LABEL=/
UUID=b1f6464f-46d7-437e-9be1-6da3ec84ae4d	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p1 LABEL=BOOT
UUID=0220-7213      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

Last edited by thewisenoob (2023-11-06 15:32:31)

seth · 2023-11-06 15:40:21

In doubt the output of "mount", but that's gonna be ext4 and as boring as it gets (btrfs, zfs or xfs would rather be candidates for such issues)

So coming back to the nvme, https://wiki.archlinux.org/title/Solid_ … leshooting
Try to disable APST and use the software IOMMU

frostschutz · 2023-11-06 16:47:52

similar case with mysterious decompression issues recently https://bbs.archlinux.org/viewtopic.php?id=289568

storage shouldn't be directly involved (if it's a fresh download, it's all in the cache). bad ram? bad cpu? it's weird!

thewisenoob · 2023-11-06 17:28:43

seth wrote:

In doubt the output of "mount", but that's gonna be ext4 and as boring as it gets (btrfs, zfs or xfs would rather be candidates for such issues)
So coming back to the nvme, https://wiki.archlinux.org/title/Solid_ … leshooting
Try to disable APST and use the software IOMMU

I tried adding nvme_core.default_ps_max_latency_us=0 and iommu=soft to my kernel params. I couldn't get GNOME to work with iommu=soft set and nvme_core.default_ps_max_latency_us=0 didn't seem to do anything either. Same error, different package than last time:

error: could not extract /usr/share/nvidia/nvoptix.bin (Zstd decompression failed: Restored data doesn't match checksum)
error: problem occurred while upgrading nvidia-utils
If you run into trouble with CUDA not being available, run nvidia-modprobe first.
If you use GDM on Wayland, you might have to run systemctl enable --now nvidia-resume.service
error: could not commit transaction
error: failed to commit transaction (transaction aborted)

I have a WD_BLACK SN850X NVMe M.2 2280 4TB if that helps. I noticed this error complains about me having the resume service disabled. I'm guessing that's not the issue, but I'll give enabling it a try again (EDIT: Doing this didn't fix my issue, as expected). I previously enabled it and the other two needed services to try Wayland, but ran into several problems so I've just been using X.

frostschutz wrote:

similar case with mysterious decompression issues recently https://bbs.archlinux.org/viewtopic.php?id=289568
storage shouldn't be directly involved (if it's a fresh download, it's all in the cache). bad ram? bad cpu? it's weird!

I noticed you have an Intel i9-13900k, which is also what I have. Possibly related?

Last edited by thewisenoob (2023-11-06 17:46:49)

seth · 2023-11-06 19:19:45

I'm guessing that's not the issue

no - isn't.

I couldn't get GNOME to work with iommu=soft set

That's odd, to say the least…
However, you could try whether the IOMMU is relevant by booting the multi-user.target w/ and w/o iommu=soft and test pacman there.

Ftr, the other thread was about multithreaded xz compression, not zstd decompression. And from the install iso.
You could still test "maxcpus=1" (but while a tolerable workaround for the installation, that's oc no solution to operate the system)

frostschutz · 2023-11-06 19:40:09

thewisenoob wrote:

I noticed you have an Intel i9-13900k, which is also what I have. Possibly related?

I don't have one and no idea.

Your error message is from libarchive (Zstd decompression failed:) and libzstd (Restored data doesn't match checksum) so it's not anything exotic.

If this fails *randomly* for intact files (otherwise pacman shouldn't even try to unpack them), it just doesn't make any sense. Something is misbehaving somewhere.

----

You can also test ram in linux using memtester (e.g. memtester 10G)

Just for kicks you can also test in parallel to give each active cpu core some work:

for i in {01..64}
do
    nice -n $((10+RANDOM%10)) unzstd --quiet --test /var/cache/pacman/pkg/suitesparse-7.3.1-1-x86_64.pkg.tar.zst &&
    printf "\n\n\n=== $i unzstd: OK ($?) ===\n\n\n" ||
    printf "\n\n\n=== $i unzstd: FAIL ($?) ===\n\n\n" & 


    nice -n $((10+RANDOM%10)) memtester 10M 2 &&
    printf "\n\n\n=== $i memtester: OK ($?) ===\n\n\n" ||
    printf "\n\n\n=== $i memtester: FAIL ($?) ===\n\n\n" & 
done | grep ===

should print like

…
=== 25 unzstd: OK (0) ===
=== 31 unzstd: OK (0) ===
=== 24 unzstd: OK (0) ===
=== 60 memtester: OK (0) ===
=== 55 memtester: OK (0) ===
=== 51 memtester: OK (0) ===
…

But I'm not sure if you could trigger your issue that way. It should either all pass, or all FAIL for unzstd if there is actually a problem with the file.

thewisenoob · 2023-11-07 02:40:15

seth wrote:

However, you could try whether the IOMMU is relevant by booting the multi-user.target w/ and w/o iommu=soft and test pacman there.

I disabled gdm, set iommu=soft in my /boot/loader/entries/arch.conf file, and restarted. Tried updating and the same error still occurred.

seth wrote:

Ftr, the other thread was about multithreaded xz compression, not zstd decompression. And from the install iso.
You could still test "maxcpus=1" (but while a tolerable workaround for the installation, that's oc no solution to operate the system)

I gave this a try as well. I disabled gdm, set maxcpus=1 in my /boot/loader/entries/arch.conf file, and restarted. Tried updating AND IT WORKED?! Obviously not a longterm solution, but glad I could at least update.

frostschutz wrote:

I don't have one and no idea.

My bad. I confused you for the OP of the thread you linked. The OP of that thread has an i9 13900k.

frostschutz wrote:

You can also test ram in linux using memtester (e.g. memtester 10G)
Just for kicks you can also test in parallel to give each active cpu core some work:
for i in {01..64}
do
    nice -n $((10+RANDOM%10)) unzstd --quiet --test /var/cache/pacman/pkg/suitesparse-7.3.1-1-x86_64.pkg.tar.zst &&
    printf "\n\n\n=== $i unzstd: OK ($?) ===\n\n\n" ||
    printf "\n\n\n=== $i unzstd: FAIL ($?) ===\n\n\n" & 


    nice -n $((10+RANDOM%10)) memtester 10M 2 &&
    printf "\n\n\n=== $i memtester: OK ($?) ===\n\n\n" ||
    printf "\n\n\n=== $i memtester: FAIL ($?) ===\n\n\n" & 
done | grep ===
should print like
…
=== 25 unzstd: OK (0) ===
=== 31 unzstd: OK (0) ===
=== 24 unzstd: OK (0) ===
=== 60 memtester: OK (0) ===
=== 55 memtester: OK (0) ===
=== 51 memtester: OK (0) ===
…
But I'm not sure if you could trigger your issue that way. It should either all pass, or all FAIL for unzstd if there is actually a problem with the file.

I gave your script a try and all passed. Here is the output:

=== 03 unzstd: OK (0) ===
=== 02 unzstd: OK (0) ===
=== 11 unzstd: OK (0) ===
=== 54 unzstd: OK (0) ===
=== 01 unzstd: OK (0) ===
=== 44 unzstd: OK (0) ===
=== 23 unzstd: OK (0) ===
=== 62 unzstd: OK (0) ===
=== 48 unzstd: OK (0) ===
=== 21 unzstd: OK (0) ===
=== 28 unzstd: OK (0) ===
=== 40 unzstd: OK (0) ===
=== 05 unzstd: OK (0) ===
=== 55 unzstd: OK (0) ===
=== 43 unzstd: OK (0) ===
=== 08 unzstd: OK (0) ===
=== 14 unzstd: OK (0) ===
=== 06 unzstd: OK (0) ===
=== 45 unzstd: OK (0) ===
=== 35 unzstd: OK (0) ===
=== 53 unzstd: OK (0) ===
=== 64 unzstd: OK (0) ===
=== 18 unzstd: OK (0) ===
=== 59 unzstd: OK (0) ===
=== 26 unzstd: OK (0) ===
=== 22 unzstd: OK (0) ===
=== 37 unzstd: OK (0) ===
=== 46 unzstd: OK (0) ===
=== 07 unzstd: OK (0) ===
=== 42 unzstd: OK (0) ===
=== 20 unzstd: OK (0) ===
=== 63 unzstd: OK (0) ===
=== 12 unzstd: OK (0) ===
=== 31 unzstd: OK (0) ===
=== 49 unzstd: OK (0) ===
=== 60 unzstd: OK (0) ===
=== 57 unzstd: OK (0) ===
=== 09 unzstd: OK (0) ===
=== 19 unzstd: OK (0) ===
=== 52 unzstd: OK (0) ===
=== 41 unzstd: OK (0) ===
=== 16 unzstd: OK (0) ===
=== 15 unzstd: OK (0) ===
=== 30 unzstd: OK (0) ===
=== 56 unzstd: OK (0) ===
=== 24 unzstd: OK (0) ===
=== 29 unzstd: OK (0) ===
=== 47 unzstd: OK (0) ===
=== 61 unzstd: OK (0) ===
=== 34 unzstd: OK (0) ===
=== 27 unzstd: OK (0) ===
=== 13 unzstd: OK (0) ===
=== 51 unzstd: OK (0) ===
=== 33 unzstd: OK (0) ===
=== 38 unzstd: OK (0) ===
=== 39 unzstd: OK (0) ===
=== 58 unzstd: OK (0) ===
=== 36 unzstd: OK (0) ===
=== 32 unzstd: OK (0) ===
=== 10 unzstd: OK (0) ===
=== 50 unzstd: OK (0) ===
=== 04 unzstd: OK (0) ===
=== 17 unzstd: OK (0) ===
=== 25 unzstd: OK (0) ===
=== 36 memtester: OK (0) ===
=== 08 memtester: OK (0) ===
=== 46 memtester: OK (0) ===
=== 57 memtester: OK (0) ===
=== 35 memtester: OK (0) ===
=== 44 memtester: OK (0) ===
=== 13 memtester: OK (0) ===
=== 14 memtester: OK (0) ===
=== 25 memtester: OK (0) ===
=== 63 memtester: OK (0) ===
=== 16 memtester: OK (0) ===
=== 52 memtester: OK (0) ===
=== 54 memtester: OK (0) ===
=== 22 memtester: OK (0) ===
=== 10 memtester: OK (0) ===
=== 48 memtester: OK (0) ===
=== 62 memtester: OK (0) ===
=== 27 memtester: OK (0) ===
=== 39 memtester: OK (0) ===
=== 60 memtester: OK (0) ===
=== 21 memtester: OK (0) ===
=== 23 memtester: OK (0) ===
=== 33 memtester: OK (0) ===
=== 15 memtester: OK (0) ===
=== 05 memtester: OK (0) ===
=== 24 memtester: OK (0) ===
=== 53 memtester: OK (0) ===
=== 42 memtester: OK (0) ===
=== 17 memtester: OK (0) ===
=== 49 memtester: OK (0) ===
=== 28 memtester: OK (0) ===
=== 51 memtester: OK (0) ===
=== 32 memtester: OK (0) ===
=== 02 memtester: OK (0) ===
=== 47 memtester: OK (0) ===
=== 37 memtester: OK (0) ===
=== 41 memtester: OK (0) ===
=== 06 memtester: OK (0) ===
=== 09 memtester: OK (0) ===
=== 12 memtester: OK (0) ===
=== 55 memtester: OK (0) ===
=== 04 memtester: OK (0) ===
=== 03 memtester: OK (0) ===
=== 45 memtester: OK (0) ===
=== 26 memtester: OK (0) ===
=== 11 memtester: OK (0) ===
=== 38 memtester: OK (0) ===
=== 19 memtester: OK (0) ===
=== 29 memtester: OK (0) ===
=== 20 memtester: OK (0) ===
=== 07 memtester: OK (0) ===
=== 01 memtester: OK (0) ===
=== 56 memtester: OK (0) ===
=== 18 memtester: OK (0) ===
=== 59 memtester: OK (0) ===
=== 61 memtester: OK (0) ===
=== 34 memtester: OK (0) ===
=== 58 memtester: OK (0) ===
=== 64 memtester: OK (0) ===
=== 50 memtester: OK (0) ===
=== 40 memtester: OK (0) ===
=== 31 memtester: OK (0) ===
=== 43 memtester: OK (0) ===
=== 30 memtester: OK (0) ===

I'm not sure if any of this implies what's going wrong. Thanks for the help you two have given me. <3

frostschutz · 2023-11-07 07:23:24

Did you run that test with maxcpus=1 or not? maxcpus=1 is a really weird workaround. So either a cpu core is misbehaving (bad cpu cache? power supply? no idea) or some obscure kernel bug? You could also try with linux-lts kernel...

Is there anything in dmesg when these Zstd errors occur?

I'm pretty much unable to help, sorry. There should be a better tool for testing CPU, including special cases like testing different class cores (P-Cores vs. E-Cores). Or maybe you could turn them off in Bios (if at all possible) and set a power limit or something, just to see if it changes anything... no idea really...

Random errors are annoying. It would be great if it was bad ram, then at least you'd know what to replace (or disable XMP profile).

seth · 2023-11-07 07:28:31

set maxcpus=1 in my /boot/loader/entries/arch.conf file, and restarted. Tried updating AND IT WORKED?!

We might have a bigger problem there…

- Do you have https://wiki.archlinux.org/title/Microcode ?

- Do you disable any mitigations?
https://raw.githubusercontent.com/torva … meters.txt / look for "mitigations="

- "ideally" the bug is in https://archlinux.org/packages/core/x86_64/libarchive/
Does downgrading to 3.7.1-1 prevent the errors?
https://archive.archlinux.org/packages/ … kg.tar.zst

thewisenoob · 2023-11-07 14:57:40

frostschutz wrote:

Did you run that test with maxcpus=1 or not? maxcpus=1 is a really weird workaround.

Yep, it worked when I tried maxcpus=1. Maybe I just got lucky. I'll give it a couple more tries just to be sure.

seth wrote:

- Do you have https://wiki.archlinux.org/title/Microcode ?

I did not, but that did not affect the problem I'm having. Still the same zstd error. I also realized my bios needed updating, but the error still occurs after updating to the latest. z9 from here: https://www.gigabyte.com/Motherboard/Z7 … rt-dl-bios

seth wrote:

- Do you disable any mitigations?

Not that I am aware of. None in my systemd-boot conf. Not sure where else I would specify them.

seth wrote:

- "ideally" the bug is in https://archlinux.org/packages/core/x86_64/libarchive/
Does downgrading to 3.7.1-1 prevent the errors?
https://archive.archlinux.org/packages/ … kg.tar.zst

I tried downgrading to 3.7.1-1 and even the last version of 3.6(3.6.2-2). I also put IgnorePkg = libarchive in /etc/pacman.conf after downgrading. Then I tried updating and the same issue occurs, unfortunately.

seth · 2023-11-07 15:12:31

Not sure where else I would specify them.

cat /proc/cmdline

I tried downgrading to 3.7.1-1 and even the last version of 3.6(3.6.2-2)

frostschutz wrote:

also try with linux-lts kernel...

thewisenoob · 2023-11-07 18:40:50

So it seems like something is up with most likely my CPU because I tried going back to an older state with timeshift and the zstd error still happens when updating.

I started doing some research about other people with instability issues with an 13900k. I found somebody recommending setting all of your P cores to a specific frequency. I set mine to 5.5Ghz. I've tested a full package reinstall multiple times using

sudo pacman -Qqn | sudo pacman -S

and it never fails anymore. I'm not sure why this is suddenly a problem for me since I built this computer about a year ago. Somehow, the cores hitting 5.8Ghz causes instability. Annoying since I'm using defaults. Oh well. Just glad I found a decent solution.

I'm going to mark this as solved. I really appreciate all of the help you two have given me. <3

seth · 2023-11-07 20:19:05

I set mine to 5.5Ghz

https://wiki.archlinux.org/title/CPU_fr … requencies ?

I'm not sure why this is suddenly a problem for me

Could it be a temperature issue?
The fans to slow or too much dust in the system?

Arch Linux

#1 2023-11-05 21:54:25

[SOLVED] zstd decompression failed: Restored data doesn't match checks

#2 2023-11-06 14:24:18

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#3 2023-11-06 14:33:32

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#4 2023-11-06 15:15:07

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#5 2023-11-06 15:28:39

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#6 2023-11-06 15:40:21

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#7 2023-11-06 16:47:52

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#8 2023-11-06 17:28:43

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#9 2023-11-06 19:19:45

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#10 2023-11-06 19:40:09

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#11 2023-11-07 02:40:15

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#12 2023-11-07 07:23:24

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#13 2023-11-07 07:28:31

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#14 2023-11-07 14:57:40

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#15 2023-11-07 15:12:31

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#16 2023-11-07 18:40:50

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

#17 2023-11-07 20:19:05

Re: [SOLVED] zstd decompression failed: Restored data doesn't match checks

Board footer