OpenSSL asn1 encoding routines / bad record mac errors - compromised?

dakralex · 2024-09-21 11:48:53

Hello there!

I've been using Arch Linux for well over four years now and haven't had a lot of issues that I couldn't solve without reading some manual pages and/or the Arch Linux Wiki.

I've built a new desktop computer at the end of last year and had no issues with the hardware at the start, except for an ACPI warning message, that some hub doesn't have any ports. I'm just guessing, but since I chose a mainboard without integrated WiFi (and there's another option with that: Gigabyte B650M GAMING X AX), I'd guess that the firmware developers didn't bother to release different versions of the firmware and the error is a false-positive. Otherwise, I have updated the firmware to the latest and here's my hardware setup:

Mainboard: Gigabyte B650M DS3H (rev. 1.3)
CPU: AMD Ryzen 9 7900X 12-Core Processor
RAM: 2x Crucial Pro 24 GB DDR5-5200 (CP24G56C46U5.M8B1)
GPU: MSI Radeon RX 570 GAMING X 4G

I've only recently run into two problems:

The first is, that I think my GPU might be dying. I've bought it off cheap and used and it ran fine for around half a year, but since around two or three months now, I get quite consistent artifacts in images and video (but not when rendering e.g. video games) and tried to debug Plasma/Kwin, X11/Wayland, mesa, vdpau, amdgpu driver and checking my cables and connections, but nothing seemed to solve the problem. Since I've booted Debian and Windows live ISOs and the problem persisted I would say it is a hardware issue. But that's not the point of this article (but if anybody has other thoughts, I'd gladly open another post, because maybe someone also has these issues).

The actual point of this post is the following: Since around that time where the GPU issues surfaced (even though I don't think they're correlated), I also had trouble using pacman. I couldn't verify the package's GPG signatures anymore and since I didn't have time to spend on the computer at that time, I haven't looked into it seriously. The problem is that the when installing them, I got the errors like the following:

error: pipewire-jack: signature from "Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>" is invalid
:: File /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).

What I have also noticed some time later is that when I'm cloning git repositories over https (which I do fairly often since I'm a developer and building many packages myself), I frequently get errors like these:

OpenSSL/3.3.2: error:06880006:asn1 encoding routines::EVP lib

OpenSSL/3.3.2: error:0A000119:SSL routines::decryption failed or bad record mac

I've already done the following, but none have solved the pacman issue nor the OpenSSL error:

Deleted all cached packages as described in /var/cache/pacman/pkg/
Updated the archlinux-keyring package and synchronized my system clock with NTP (which I've already setup but just to be sure) as described at [0]
Repopulating the pacman keyring in various ways as described at [1] and [2]
Running "pacman-key --refresh-keys" as root
Changing the key server to "hkp://keyserver.ubuntu.com" for my personal and the root user
Reinstalling all SSL/TLS/CA-related packages (e.g. openssl, openssl-1.1, gnutls, libressl
Reinstalling ca-certificates and running update-ca-trust (also tried removing the folder completely and then repopulating the certificates)
Disallowing any packet redirects for IPv4 and IPv6 in the kernel (e.g. net.ipv4.conf.all.accept_redirects = 0)
Reinstalling all packages with: pacman -Qqn | pacman -S -
Running memtest86+ for 12 hours without any errors
Running a full S.M.A.R.T. test for ~1.5 hours without any errors logged

My last resort has been that I've used pacman with "SigLevel = Never", where I can install and update packages without any troubles - of course.

I'm not sure what I can try anymore except reinstalling the system. I've also tried booting into Debian and Windows Live ISOs and I couldn't recreate that problem there, so I'm ruling out hardware errors or that my router/modem and/or mainboard is compromised. It is quite frustrating that I cannot use HTTPS and pacman reliably on my computer and that causes quite a few issues (e.g. I also noticed that in Firefox I often get "Image corrupt or truncated." on images that are sent over HTTPS).

I'm sorry for the long post and I'd gladly provide any more information if needed. Thanks for anybody taking a look at this, I'd appreciate some help very much!

Cheers!

[0] https://wiki.archlinux.org/title/Pacman … _regularly
[1] https://wiki.archlinux.org/title/Pacman … l_the_keys
[2] https://wiki.archlinux.org/title/Pacman … mport_keys

Last edited by dakralex (2024-09-21 13:21:10)

mithrial · 2024-09-21 11:50:58

Is your time set correctly? Also, could it possibly be DNS?

dakralex · 2024-09-21 12:13:40

mithrial wrote:

Is your time set correctly? Also, could it possibly be DNS?

Yes, I've setup my system to regularly synchronize with a NTP server. When doing it manually those are the results:

$ ntpd -gq
21 Sep 13:55:51 ntpd[75393]: ntpd 4.2.8p18@1.4062-o Thu May 30 16:14:20 UTC 2024 (1): Starting
21 Sep 13:55:51 ntpd[75393]: Command line: ntpd -gq
21 Sep 13:55:51 ntpd[75393]: ----------------------------------------------------
21 Sep 13:55:51 ntpd[75393]: ntp-4 is maintained by Network Time Foundation,
21 Sep 13:55:51 ntpd[75393]: Inc. (NTF), a non-profit 501(c)(3) public-benefit
21 Sep 13:55:51 ntpd[75393]: corporation.  Support and training for ntp-4 are
21 Sep 13:55:51 ntpd[75393]: available at https://www.nwtime.org/support
21 Sep 13:55:51 ntpd[75393]: ----------------------------------------------------
21 Sep 13:55:51 ntpd[75393]: DEBUG behavior is enabled - a violation of any diagnostic assertion will cause ntpd to abort
21 Sep 13:55:51 ntpd[75393]: proto: precision = 0.030 usec (-25)
21 Sep 13:55:51 ntpd[75393]: basedate set to 2024-05-18
21 Sep 13:55:51 ntpd[75393]: gps base set to 2024-05-19 (week 2315)
21 Sep 13:55:51 ntpd[75393]: initial drift restored to 21.943054
21 Sep 13:55:51 ntpd[75393]: unable to bind to wildcard address :: - another process may be running - EXITING
$ hwclock -w
$ timedatectl
               Local time: Sat 2024-09-21 13:56:00 CEST
           Universal time: Sat 2024-09-21 11:56:00 UTC
                 RTC time: Sat 2024-09-21 11:56:00
                Time zone: Europe/Vienna (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Unfortunately, this doesn't solve the problem. As ntpd is already running, I restarted the systemd unit again and got the following in `systemctl status ntpd.service`, which is strange, but the system time itself is accurate.

kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

Also, I'm using the Google DNS server:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888

I've just to also try disabling IPv6 and verified that my DNS cache will use the IPv4 addresses, but I still get both the pacman as well as the OpenSSL errors.

Last edited by dakralex (2024-09-21 12:14:34)

seth · 2024-09-21 14:01:49

I would say it is a hardware issue.

Do thesy show up in screenshots?
Do you have a photo?

My last resort has been that I've used pacman with "SigLevel = Never"

Reinstalling all packages with: pacman -Qqn | pacman -S -

WAHHHHHHHH…… AHHHHHHHHH… AAAHHHHHHHhhhhh…… AAaaaaaahhhhhhhh … aahhhhhh
Don't do that.

First, let's check this dubious "heftig" guy…

pacman-key -l heftig

Then the package

md5sum /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64*

Then openssl

pacman -Qikk openssl glibc ca-certificates ca-certificates-mozilla

Then why do you have libressl installed??

dakralex · 2024-09-21 16:05:49

seth wrote:

Do thesy show up in screenshots?
Do you have a photo?

Yes, for example here for video [0] and here for images [1]. They are quite consistent and have the same "form and shape" when booting a Debian 12.7 Live ISO (running a Linux 6.1 LTS kernel) and the latest Hiren's BootCD.

seth wrote:

WAHHHHHHHH…… AHHHHHHHHH… AAAHHHHHHHhhhhh…… AAaaaaaahhhhhhhh … aahhhhhh
Don't do that.

It sounds like you lost your faith in humanity here, I'm truly sorry :'). Could you explain to me why I should never do this? Is it because it completely broke the chain of trust in package signatures?

Here are the outputs for the commands you mentioned:

$ pacman-key -l heftig
pub   rsa2048 2011-08-25 [SC]
      8218F88849AAC522E94CF470A5E9288C4FA415FA
uid           [marginal] Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
sub   rsa2048 2011-08-25 [E]
sub   rsa4096 2019-05-08 [A]
sub   ed25519 2019-05-08 [A]

pub   ed25519 2023-12-11 [SC]
      83BC8889351B5DEBBB68416EB8AC08600F108CDF
uid           [  undef ] Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
uid           [  full  ] Jan Alexander Steffens (heftig) <heftig@archlinux.org>
sub   ed25519 2023-12-11 [A]
sub   cv25519 2023-12-11 [E]

pub   ed25519 2020-05-11 [SC]
      A2FF3A36AAA56654109064AB19802F8B0D70FC30
uid           [ unknown] Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
uid           [  full  ] Jan Alexander Steffens (heftig) <heftig@archlinux.org>
uid           [ unknown] Jan Alexander Steffens (heftig) <jan.steffens@ltnglobal.com>
sub   ed25519 2020-05-11 [A]
sub   ed25519 2020-05-11 [S]
sub   cv25519 2020-05-11 [E]

This is the same output as on my laptop that doesn't have these issues (though the 2023 pubkey entry is at the bottom there).

$ md5sum /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst
288d712b45fb3f2850b992f3c506b259  /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst

Seems as expected and is the same as it is on archive.archlinux.org at [2].

$ pacman -Qikk openssl glibc ca-certificates ca-certificates-mozilla
Name            : openssl
Version         : 3.3.2-1
Description     : The Open Source toolkit for Secure Sockets Layer and Transport Layer Security
Architecture    : x86_64
URL             : https://www.openssl.org
Licenses        : Apache-2.0
Groups          : None
Provides        : libcrypto.so=3-64  libssl.so=3-64
Depends On      : glibc
Optional Deps   : ca-certificates [installed]
                  perl [installed]
Required By     : bind  coreutils  cryptsetup  curl  dhcpcd  f5fpc  git  gnustep-base  grub-customizer  gst-plugins-bad  hexchat  kmod  krb5
                  lib32-openssl  libarchive  libdatachannel  libevent  libgit2  libimobiledevice  libnvme  libquotient  libsasl  libshout  libssh
                  libssh2  libtorrent-rasterbar  libzip  lynx  neon  nmap  ntp  open-iscsi  open-isns  openssh  openvpn  opusfile  partimage  perf
                  pkcs11-helper  postgresql  postgresql-libs  ppp  python  python2-bin  qbittorrent  qpdf  qt6-base  rsync  sequoia-sq  socat  srt
                  sudo  systemd  tcpdump  testdisk  testssl.sh  tpm2-tss  unarchiver  wpa_supplicant
Optional For    : apr-util  archiso  qca-qt5  qca-qt6
Conflicts With  : None
Replaces        : openssl-perl  openssl-doc
Installed Size  : 10.98 MiB
Packager        : Pierre Schmitz <pierre@archlinux.org>
Build Date      : Tue 03 Sep 2024 06:32:59 PM CEST
Install Date    : Sat 14 Sep 2024 04:51:29 PM CEST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : SHA-256 Sum

openssl: 6147 total files, 0 altered files
Name            : glibc
Version         : 2.40+r16+gaa533d58ff-2
Description     : GNU C Library
Architecture    : x86_64
URL             : https://www.gnu.org/software/libc
Licenses        : GPL-2.0-or-later  LGPL-2.1-or-later
Groups          : None
Provides        : None
Depends On      : linux-api-headers>=4.10  tzdata  filesystem
Optional Deps   : gd: for memusagestat [installed]
                  perl: for mtrace [installed]
Required By     : a52dec  aalib  abseil-cpp  [...redacted because of sheer size of the list...]  zlib  zopfli  zps  zstd zxing-cpp
Optional For    : tzdata
Conflicts With  : None
Replaces        : None
Installed Size  : 47.64 MiB
Packager        : Frederik Schwan <freswa@archlinux.org>
Build Date      : Mon 05 Aug 2024 10:00:52 PM CEST
Install Date    : Fri 13 Sep 2024 11:57:22 PM CEST
Install Reason  : Installed as a dependency for another package
Install Script  : Yes
Validated By    : SHA-256 Sum

backup file: glibc: /etc/locale.gen (Modification time mismatch)
backup file: glibc: /etc/locale.gen (Size mismatch)
backup file: glibc: /etc/locale.gen (SHA256 checksum mismatch)
glibc: 1614 total files, 0 altered files
Name            : ca-certificates
Version         : 20240618-1
Description     : Common CA certificates - default providers
Architecture    : any
URL             : https://src.fedoraproject.org/rpms/ca-certificates
Licenses        : GPL-2.0-or-later
Groups          : None
Provides        : None
Depends On      : ca-certificates-mozilla
Optional Deps   : None
Required By     : curl  mono  neon  python-certifi  python-requests  qca-qt5  qca-qt6
Optional For    : lib32-openssl  libressl  neomutt  openssl  wget
Conflicts With  : ca-certificates-cacert<=20140824-4
Replaces        : ca-certificates-cacert<=20140824-4
Installed Size  : 0.00 B
Packager        : Jan Alexander Steffens (heftig) <heftig@archlinux.org>
Build Date      : Tue 18 Jun 2024 08:36:40 PM CEST
Install Date    : Sat 14 Sep 2024 04:51:29 PM CEST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : SHA-256 Sum

ca-certificates: 0 total files, 0 altered files
Name            : ca-certificates-mozilla
Version         : 3.104-1
Description     : Mozilla's set of trusted CA certificates
Architecture    : x86_64
URL             : https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS
Licenses        : MPL-2.0
Groups          : None
Provides        : None
Depends On      : ca-certificates-utils>=20181109-3
Optional Deps   : None
Required By     : ca-certificates
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 1069.00 KiB
Packager        : Jan Alexander Steffens (heftig) <heftig@archlinux.org>
Build Date      : Sat 31 Aug 2024 01:36:20 AM CEST
Install Date    : Fri 13 Sep 2024 11:57:30 PM CEST
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : SHA-256 Sum

ca-certificates-mozilla: 5 total files, 0 altered files

I cannot see any wrong with that output, is there something that I'm missing here?

For the libressl part...I'm actually not sure why I did that. Nothing is depending on that package, so I uninstalled it again.

[0] https://imgur.com/a/iT5x9xb
[1] https://imgur.com/a/aYwG3mK
[2] https://archive.archlinux.org/packages/p/pipewire-jack/

Last edited by dakralex (2024-09-21 16:10:16)

seth · 2024-09-21 16:29:58

it completely broke the chain of trust in package signaturesit completely broke the chain of trust in package signatures

seth wrote:

md5sum /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64*

Where's the signature?

https://imgur.com/a/aYwG3mK is 404 https://imgur.com/a/iT5x9xb looks more like a render than a signal issue (and I assume shows up on screensots?)

dakralex · 2024-09-21 16:52:06

seth wrote:

Where's the signature?

Oh, I missed that, sorry! Here's the full output. It's still the same md5sums as the one on archive.archlinux.org for me.

$ md5sum /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst*
288d712b45fb3f2850b992f3c506b259  /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst
70495e608141cbc0da953fbcda43c2e3  /var/cache/pacman/pkg/pipewire-jack-1:1.2.4-1-x86_64.pkg.tar.zst.sig

seth wrote:

https://imgur.com/a/aYwG3mK is 404 https://imgur.com/a/iT5x9xb looks more like a render than a signal issue (and I assume shows up on screensots?)

Not sure why the first link was taken down, but it was just a random image search (of cats) with the same artifacts as in the video. But yes, the artifacts show up on screenshots on either Wayland and X11, and also on Hiren's Boot CD and a Debian 12.7 Live ISO (both in Firefox and Chromium-based browers like Chrome and Edge) to make sure it isn't because of my particular software setup - but I'm very open for suggestions. Unfortunately, I don't have another graphics card to check if it's happening with those too. I'll have to ask around if some friend would be generous enough for them to take one of their graphic cards over.

Here's a screenshot [0] that is also how my KDE Plasma 6 desktop sometimes looks like (vanilla, haven't installed/configured any themes as far as I'm aware).

[0] https://imgur.com/a/NfJtZcE

seth · 2024-09-21 21:55:10

So the signature is there and the hashes are correct.
heftig's key is there and correct
You've reset the keyring (and the master key)

Does that very package still generate the error?

pacman -S pipewire-jack

kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

You say the time is correct, but dows that also hold for the RTC?

Arch Linux

#1 2024-09-21 11:48:53

OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#2 2024-09-21 11:50:58

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#3 2024-09-21 12:13:40

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#4 2024-09-21 14:01:49

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#5 2024-09-21 16:05:49

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#6 2024-09-21 16:29:58

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#7 2024-09-21 16:52:06

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

#8 2024-09-21 21:55:10

Re: OpenSSL asn1 encoding routines / bad record mac errors - compromised?

Board footer