When you blame a core library, but it's actually a physical bit-flip.

babaliaris · Yesterday 12:47:17

Hey everyone,

Just wanted to share a recent hardware debugging story and see what strange RAM failure experiences others on Arch have run into.

System Specs:
CPU : AMD Ryzen 7 5800X3D
Motherboard: MSI B550 (running Combo Strike 3 undervolt)
RAM : 32GB (2x16GB) Corsair Vengeance RGB Pro DDR4-3600 (XMP Profile)

Symptoms:
Out of nowhere, random user-space applications started segfaulting intermittently. Because I was running an undervolt (Combo Strike 3) on my CPU, my initial reaction was to blame either CPU instability or a software regression in a recently updated core system library.

What I tried:
1) CMOS clear by shorting the JBAT1 pins.
2) Fired up memtester inside Arch:

[babaliaris@babaliaris ~]$ sudo memtester 24G 1
memtester version 4.7.1 (64-bit)
Copyright (C) 2001-2024 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 24576MB (25769803776 bytes)
got  24576MB (25769803776 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : testing   1FAILURE: possible bad address line at offset 0x0000000152614ee8.
Skipping to next test...
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : testing  27FAILURE: 0xffffffffffdfffff != 0xffffffffffffffff at offset 0x0000000152614ee8.
  Block Sequential    : testing 160FAILURE: 0xa0a0a0a0a080a0a0 != 0xa0a0a0a0a0a0a0a0 at offset 0x0000000152614ee8.
  Checkerboard        : testing   0FAILURE: 0x5555555555755555 != 0x5555555555555555 at offset 0x0000000152613e60.
FAILURE: 0x5555555555755555 != 0x5555555555555555 at offset 0x0000000152614220.
  Bit Spread          : testing   1FAILURE: 0x000000000020000a != 0x000000000000000a at offset 0x0000000152613e60.
  Bit Flip            : testing   3FAILURE: 0xffffffffffdffffe != 0xfffffffffffffffe at offset 0x0000000152614ee8.
  Walking Ones        : ok
  Walking Zeroes      : testing  21FAILURE: 0x0000000000000000 != 0x0000000000200000 at offset 0x0000000152614ee8.
  8-bit Writes        : |FAILURE: 0x7ffdb845b7f9509d != 0x7ffdb845b7b9509d at offset 0x0000000152615390.
  16-bit Writes       : \FAILURE: 0x53fa3df7aee93fc0 != 0x53fa3df7aec93fc0 at offset 0x0000000152614220.

Done.


[babaliaris@babaliaris ~]$ fastfetch
                  -`                     babaliaris@babaliaris
                 .o+`                    ---------------------
                `ooo/                    OS: Arch Linux x86_64
               `+oooo:                   Host: MS-7C91 (1.0)
              `+oooooo:                  Kernel: Linux 7.2.4-arch1-2
              -+oooooo+:                 Uptime: 40 mins
            `/:-:++oooo+:                Packages: 16 (flatpak-system), 6 (flatpak-user), 1181 (pacman)
           `/++++/+++++++:               Shell: bash 5.3.15
          `/++++++++++++++:              Display (DELL G3223D): 2560x1440 in 31", 165 Hz [External]
         `/+++ooooooooooooo/`            Desktop Environment: KDE Plasma 6.7.5
        ./ooosssso++osssssso+`           Window Manager: KWin (Wayland)
       .oossssso-````/ossssss+`          WM Theme: Breeze
      -osssssso.      :ssssssso.         Theme: oxygen (Dark) [Qt], Breeze-Dark [GTK2], Breeze [GTK3]
     :osssssss/        osssso+++.        Icons: oxygen [Qt], oxygen [GTK2/3/4]
    /ossssssss/        +ssssooo/-        Font: Noto Sans (10pt) [Qt], Noto Sans (10pt) [GTK2/3/4]
  `/ossssso+/:-        -:/+osssso+-      Cursor: Oxygen_Black (24px)
 `+sso+:-`                 `.-/+oso:     Terminal: konsole 26.8.1
`++:.                           `-/+/    Terminal Font: JetBrainsMono Nerd Font (10pt)
.`                                 `/    CPU: AMD Ryzen 7 5800X3D (16) @ 4.55 GHz
                                         GPU: AMD Radeon RX 9060 XT [Discrete]
                                         Memory: 3.67 GiB / 31.26 GiB (12%)
                                         Swap: 0 B / 32.00 GiB (0%)
                                         Disk (/): 1.29 TiB / 1.78 TiB (72%) - ext4
                                         Disk (/home/babaliaris/mnt/kingston_M.2_500GB): 216.76 GiB / 457.38 GiB (47%) - ext4
                                         Local IP (wlo1): 192.168.68.109/24
                                         Locale: en_US.UTF-8



[babaliaris@babaliaris ~]$ sudo dmidecode -t memory | grep -E "Speed|Configured Clock Speed|Part Number|Manufacturer"
[sudo] password for babaliaris:
        Speed: 2133 MT/s
        Manufacturer: Unknown
        Part Number: CMW32GX4M2D3600C18
        Configured Memory Speed: 2133 MT/s
        Speed: 2133 MT/s
        Manufacturer: Unknown
        Part Number: CMW32GX4M2D3600C18
        Configured Memory Speed: 2133 MT/s



[babaliaris@babaliaris ~]$ sudo dmidecode -t bios | grep -E "Vendor|Version|Release Date"
[sudo] password for babaliaris:
        Vendor: American Megatrends International, LLC.
        Version: 1.L1
        Release Date: 07/17/2026

The mem-test was much worse before the CMOS clearing.

I requested an RMA warranty from Corsair and got it within 5 minutes. Great support!

I hope this is the only problem with my hardware... It's too expensive to buy new hardware these days...

By the way, I updated my system yesterday when I was not aware of this problem. Could this have messed up anything in
the SSD drive, like core binary files or anything?

I'm still using the system. Sometimes, Firefox tabs crash randomly, or something else will crash, but I need to keep using the system
because I'm reading/preparing for exams.

frostschutz · Yesterday 13:20:09

Could this have messed up anything in the SSD drive, like core binary files or anything?

Verify files if possible.

If the bad address is in a fixed region, you can use badram (in Grub) or memmap= kernel parameter so that region won't be used. Kernel also has memtest= parameter which is a very simple early memtest and if it encounters any errors there it will mark those regions as bad too.

babaliaris · Yesterday 13:45:04

[babaliaris@babaliaris ~]$ sudo pacman -Qkk $(pacman -Qsq) | grep -v " 0 altered files"
[sudo] password for babaliaris: 
Sorry, try again.
[sudo] password for babaliaris: 
warning: amd-ucode: /boot/amd-ucode.img (Permissions mismatch)
warning: cups: /etc/cups/classes.conf (Permissions mismatch)
warning: cups: /etc/cups/printers.conf (Permissions mismatch)
warning: cups: /etc/cups/subscriptions.conf (Permissions mismatch)
amd-ucode: 17 total files, 1 altered file
backup file: cups: /etc/cups/printers.conf (Modification time mismatch)
backup file: cups: /etc/cups/printers.conf (Size mismatch)
backup file: cups: /etc/cups/printers.conf (SHA256 checksum mismatch)
backup file: cups: /etc/cups/subscriptions.conf (Modification time mismatch)
backup file: cups: /etc/cups/subscriptions.conf (Size mismatch)
backup file: cups: /etc/cups/subscriptions.conf (SHA256 checksum mismatch)
cups: 947 total files, 3 altered files
backup file: filesystem: /etc/fstab (Modification time mismatch)
backup file: filesystem: /etc/fstab (Size mismatch)
backup file: filesystem: /etc/fstab (SHA256 checksum mismatch)
backup file: filesystem: /etc/group (Modification time mismatch)
backup file: filesystem: /etc/group (Size mismatch)
backup file: filesystem: /etc/group (SHA256 checksum mismatch)
backup file: filesystem: /etc/gshadow (Modification time mismatch)
backup file: filesystem: /etc/gshadow (Size mismatch)
backup file: filesystem: /etc/gshadow (SHA256 checksum mismatch)
backup file: filesystem: /etc/nsswitch.conf (Modification time mismatch)
backup file: filesystem: /etc/nsswitch.conf (Size mismatch)
backup file: filesystem: /etc/nsswitch.conf (SHA256 checksum mismatch)
backup file: filesystem: /etc/passwd (Modification time mismatch)
backup file: filesystem: /etc/passwd (Size mismatch)
backup file: filesystem: /etc/passwd (SHA256 checksum mismatch)
backup file: filesystem: /etc/resolv.conf (Modification time mismatch)
backup file: filesystem: /etc/resolv.conf (Size mismatch)
backup file: filesystem: /etc/resolv.conf (SHA256 checksum mismatch)
backup file: filesystem: /etc/shadow (Modification time mismatch)
backup file: filesystem: /etc/shadow (Size mismatch)
backup file: filesystem: /etc/shadow (SHA256 checksum mismatch)
backup file: filesystem: /etc/shells (Modification time mismatch)
backup file: filesystem: /etc/shells (Size mismatch)
backup file: filesystem: /etc/shells (SHA256 checksum mismatch)
backup file: filesystem: /etc/subgid (Modification time mismatch)
backup file: filesystem: /etc/subgid (Size mismatch)
backup file: filesystem: /etc/subgid (SHA256 checksum mismatch)
backup file: filesystem: /etc/subuid (Modification time mismatch)
backup file: filesystem: /etc/subuid (Size mismatch)
backup file: filesystem: /etc/subuid (SHA256 checksum mismatch)
backup file: glibc: /etc/locale.gen (Modification time mismatch)
backup file: glibc: /etc/locale.gen (Size mismatch)
backup file: glibc: /etc/locale.gen (SHA256 checksum mismatch)
backup file: grub: /etc/default/grub (Modification time mismatch)
backup file: grub: /etc/default/grub (Size mismatch)
backup file: grub: /etc/default/grub (SHA256 checksum mismatch)
backup file: grub: /etc/grub.d/40_custom (Modification time mismatch)
backup file: grub: /etc/grub.d/40_custom (Size mismatch)
backup file: grub: /etc/grub.d/40_custom (SHA256 checksum mismatch)
warning: libutempter: /usr/lib/utempter/utempter (GID mismatch)
warning: libutempter: /usr/lib/utempter/utempter (Permissions mismatch)
warning: libvlc: /usr/lib/vlc/plugins/plugins.dat (Modification time mismatch)
warning: libvlc: /usr/lib/vlc/plugins/plugins.dat (Size mismatch)
warning: libvlc: /usr/lib/vlc/plugins/plugins.dat (SHA256 checksum mismatch)
libutempter: 20 total files, 1 altered file
libvlc: 120 total files, 1 altered file
warning: openssl: /etc/ssl/private (Permissions mismatch)
warning: passim: /var/lib/passim (UID mismatch)
warning: passim: /var/lib/passim (GID mismatch)
warning: passim: /var/lib/passim/data (UID mismatch)
warning: passim: /var/lib/passim/data (GID mismatch)
openssl: 6752 total files, 1 altered file
backup file: pacman: /etc/pacman.conf (Modification time mismatch)
backup file: pacman: /etc/pacman.conf (Size mismatch)
backup file: pacman: /etc/pacman.conf (SHA256 checksum mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Modification time mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (Size mismatch)
backup file: pacman-mirrorlist: /etc/pacman.d/mirrorlist (SHA256 checksum mismatch)
backup file: pam: /etc/security/limits.conf (Modification time mismatch)
backup file: pam: /etc/security/limits.conf (Size mismatch)
backup file: pam: /etc/security/limits.conf (SHA256 checksum mismatch)
passim: 92 total files, 2 altered files
backup file: reflector: /etc/xdg/reflector/reflector.conf (Modification time mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (Size mismatch)
backup file: reflector: /etc/xdg/reflector/reflector.conf (SHA256 checksum mismatch)
backup file: sudo: /etc/sudoers (Modification time mismatch)
backup file: sudo: /etc/sudoers (Size mismatch)
backup file: sudo: /etc/sudoers (SHA256 checksum mismatch)
warning: systemd: /var/log/journal (GID mismatch)
backup file: systemd: /etc/systemd/user.conf (Modification time mismatch)
backup file: systemd: /etc/systemd/user.conf (Size mismatch)
backup file: systemd: /etc/systemd/user.conf (SHA256 checksum mismatch)
systemd: 1922 total files, 1 altered file
backup file: texlive-basic: /etc/texmf/web2c/fmtutil.cnf (Modification time mismatch)
backup file: texlive-basic: /etc/texmf/web2c/fmtutil.cnf (Size mismatch)
backup file: texlive-basic: /etc/texmf/web2c/fmtutil.cnf (SHA256 checksum mismatch)

I do not see any important binaries here. Most of these configuration files were altered by me (sudoers, /etc/default grub, systemd users, fstab all manually edited by me), so I believe this is why they mismatch.

I will try the kernel parameter as well. The new RAM kit will be here in 1-2 days.

babaliaris · Yesterday 21:27:27

By the way, I found the faulty stick, and I decided to run my system with the healthy 16GB one alone until the new set arrives.

One week ago, I installed my new CPU, Ryzen 5800X3D, and I thought I had damaged something or that the new CPU had a problem.
I was very devastated. Thank god Corsair was excellent with support!!!

I had a similar issue with my DELL Monitor, and DELL had great support as well. They replaced it immediately and
I had no problems since.

In 2014, I had another computer at that time that was assembled by ESHOP, a famous company in Greece, on "pick your hardware and we build it for you". I left for 2 months for vacation, and when I returned back my PC would not boot. Turns out it was the RAM.

I'm pretty sure I'm one of the most unlucky persons on Earth, getting the 1% defective hardware out there. I'm cursed! Thank god I had great warranty support.

Arch Linux

#1 Yesterday 12:47:17

When you blame a core library, but it's actually a physical bit-flip.

#2 Yesterday 13:20:09

Re: When you blame a core library, but it's actually a physical bit-flip.

#3 Yesterday 13:45:04

Re: When you blame a core library, but it's actually a physical bit-flip.

#4 Yesterday 21:27:27

Re: When you blame a core library, but it's actually a physical bit-flip.

Board footer