Indefinite Instability ~2 years in [SOLVED]

covid19 · 2022-09-02 20:56:43

First of all I love Arch, the way you put it together yourself, no extra crap, very easy to customize and rebuild packages, I cannot get this functionality from another distro with this level of ease.
Having said that, I've been running Arch for about 2 years now, I have yet to see an uptime on my system more than 30 days. Maybe ONCE in 2 years. It's infuriating how unstable my entire experience is, I never know which action I attempt is going to completely make the system take a crap. It's embarassing in front of other people, friends. These days I don't say a single word about my distro, just STFU and hope that it works.
I'm not having any repeatable type of hard or soft lockups, each time it's something a little different.
This morning I come to my laptop, attempt to enter password at the screensaver, and nothing happens at all--doesn't tell me incorrect password, just flat out nothing. Then I switch over to virtual terminals, none of them are able to login, and are experiencing SEVERE lag. 5hrs earlier I had gone to bed and everything was working, no updates install behind my back or automatically. I wasted 3 hours trying to get that session going. I took pictures but the virtual terminals displayed messages such as:

sudo: timed out reading password

-- unable to start/stop lightdm or interact with the gui/X

systemctl suspend
Failed to set wall message, ignoring: Transport endpoint is not connected
Failed to suspend system via logind: Transport endpoint is not connected
Failed to start suspend.target: Transport endpoint is not connected

At a point it wasn't even accepting my password, took 45mins to issue a 'passwd' as root, not that it helped much.
Unable to suspend the system, unable to interact with anything. Of course I had to hard shut down the system and start back up again.

See next time it will be something else, I've been silently dealing with this for 2 years, thinking oh maybe I had some bleeding edge update. But after 2 YEARS?
Are there any hints as to why my Arch stability is absolute piss poor? Sarcasm aside, it's literally like using Windows ME, I don't know which action is going to completely cause the system to diahreahh all over the place. I only have around 700 packages, this is not a heavy install by any means. Everything works well until all of a sudden it doesn't, definitely not a hardware/CPU/RAM issue as I can have 20 days of stability before it shows its colors. Moreover, before I installed Arch I ran Windows 10 and Hackintosh, neither OS displayed this level of unreliability. I want to love Arch, but it's becoming very difficult when stability is this poor.

Please help, I'm not interested in switching distros, there has to be a reason Arch is so incredibly bad on my laptop.
It's nothing bleeding edge whatsoever, HP Elitebook 8470p, i7-3xxx, Ivy bridge. Integrated HD 4000.

Last edited by covid19 (2022-10-27 00:26:39)

Slithery · 2022-09-02 21:13:11

I'd assume that you are either running partial upgrades or it's a hardware error.
I've been running this same installation for 15 years now having migrated it to 4 different machines every time I got new hardware and never had any issues...

covid19 · 2022-09-02 21:29:07

No partial upgrades, and I don't experience the same issues under Windows.
The only updates I don't take, are

linux linux-headers

, because I don't want to immediately reboot. Everytime it takes a dump on me I end up running a full pacman -Syu letting it upgrade the kernel because at that point my uptime is lost anyway. I'm not trying to say I want server uptimes with thousands of days, but I'm not exaggerating 30 days max. It saddens me incredibly that I have a macbook a few feet away and the uptime is about 1200 days. Makes me cringe

Last edited by covid19 (2022-09-02 21:31:16)

jasonwryan · 2022-09-02 21:39:20

Post a journal from a crashed or broken session.

covid19 · 2022-09-02 23:00:23

Would I obtain that via journalctl? It seems like it only goes as far back as current startup, I looked in /etc/systemd/journald.conf and see

[Journal]
#Storage=auto

Do I need to uncomment this? Or can I obtain this information in a different way?

Thanks for the help

loqs · 2022-09-02 23:06:28

Quick check the following will show all messages in the journal from the unit systemd-journald.service

# journalctl -u systemd-journald.service

Is there one similar to "Time spent on flushing to /var/log/journal/...."

ewaller · 2022-09-02 23:20:07

covid19 wrote:

[Journal]
#Storage=auto
Do I need to uncomment this? Or can I obtain this information in a different way?
Thanks for the help

No, although it is commented, the commented values in that file represent the default values.

seth · 2022-09-03 07:23:23

Do you shut down/reboot the system with the power button?
Cause that'll prevent the journal from being stored and get you into trouble very fast.

Otherwise

grep -ri Storage /{usr/lib,etc,run}/systemd/journald.conf*

but the symptoms quite frankly yell that the HW is toast and I'd start w/ a "smartctl -a" …

sudo journalctl | curl -F 'file=@-' 0x0.st

now to add some data that might elevate this thread over FUD.

covid19 · 2022-09-05 21:59:22

seth wrote:

Do you shut down/reboot the system with the power button?
Cause that'll prevent the journal from being stored and get you into trouble very fast

Nope @ power button, only to force shutdown when system takes a dump.

seth wrote:

Otherwise grep -ri Storage /{usr/lib,etc,run}/systemd/journald.conf* but the symptoms quite frankly yell that the HW is toast and I'd start w/ a "smartctl -a" …
now to add some data that might elevate this thread over FUD.

j@archbook ~]$ grep -ri Storage /{usr/lib,etc,run}/systemd/journald.conf*
grep: /usr/lib/systemd/journald.conf*: No such file or directory
/etc/systemd/journald.conf:#Storage=auto
grep: /run/systemd/journald.conf*: No such file or directory

Here's the output of the given link.
http://0x0.st/ofHI.txt

Thanks!

Last edited by covid19 (2022-09-24 20:04:46)

jasonwryan · 2022-09-05 22:06:58

Aug 21 17:12:39 archbook kernel: Linux version 5.8.2-arch1-1

Really?

loqs · 2022-09-05 22:33:21

Aug 21 17:12:39 archbook systemd-journald[273]: System Journal (/var/log/journal/96e324b9a2db417d8f15977c1475c4af) is 539.2M, max 4.0G, 3.4G free.

The journal is being written to persistent storage and from a count of the string "-- Boot" the journal you posted covers over a hundred boots.
Edit:
This is the most recent boot warning 9.5M http://0x0.st/ofXt.txt
Selected extracts
We have four times

Sep 02 21:38:44 archbook kernel: ------------[ cut here ]------------
Sep 02 21:38:44 archbook kernel: WARNING: CPU: 0 PID: 54923 at net/wireless/sme.c:766 __cfg80211_connect_result+0x3b4/0x3e0 [cfg80211]
Sep 02 21:38:44 archbook kernel: Modules linked in: vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock tun xt_tcpudp rfcomm ip6table_filter ip6_tables iptable_filter cmac algif_hash algif_skcipher af_alg bnep btusb btrtl btbcm btintel btmtk bluetooth ecdh_generic snd_hda_codec_hdmi snd_ctl_led snd_hda_codec_idt mousedev snd_hda_codec_generic ledtrig_audio intel_rapl_msr snd_hda_intel intel_rapl_common vmnet(OE) snd_intel_dspcfg snd_intel_sdw_acpi x86_pkg_temp_thermal snd_hda_codec intel_powerclamp snd_hda_core i915 iTCO_wdt hp_wmi sparse_keymap coretemp intel_pmc_bxt snd_hwdep drm_buddy wl(POE) mei_hdcp mei_pxp iTCO_vendor_support platform_profile wmi_bmof rapl joydev snd_pcm ttm mei_wdt intel_cstate drm_display_helper snd_timer intel_uncore mei_me cfg80211 snd cec psmouse lpc_ich rfkill soundcore e1000e intel_gtt mei wmi hp_accel wireless_hotkey mac_hid video lis3lv02d vmmon(OE) vmw_vmci crypto_user fuse bpf_preload ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2
Sep 02 21:38:44 archbook kernel:  dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod crct10dif_pclmul crc32_pclmul serio_raw crc32c_intel ghash_clmulni_intel atkbd libps2 vivaldi_fmap aesni_intel crypto_simd cryptd sdhci_pci cqhci sdhci xhci_pci mmc_core xhci_pci_renesas i8042 serio vfat fat
Sep 02 21:38:44 archbook kernel: Unloaded tainted modules: hv_vmbus():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 asus_ec_sensors():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 asus_ec_sensors():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1
Sep 02 21:38:44 archbook kernel: CPU: 0 PID: 54923 Comm: kworker/u16:0 Tainted: P          IOE     5.19.6-arch1-1 #1 1307568ef808cb599ee891e507344feaf47e0ed7
Sep 02 21:38:44 archbook kernel: Hardware name: Hewlett-Packard HP EliteBook 8470p/179B, BIOS 68ICF Ver. F.68 07/13/2017
Sep 02 21:38:44 archbook kernel: Workqueue: cfg80211 cfg80211_event_work [cfg80211]
Sep 02 21:38:44 archbook kernel: RIP: 0010:__cfg80211_connect_result+0x3b4/0x3e0 [cfg80211]
Sep 02 21:38:44 archbook kernel: Code: 0f 84 57 fe ff ff 48 8b b8 80 01 00 00 e8 a4 db 7a db 48 8b 7b 78 e8 9b db 7a db 48 c7 43 78 00 00 00 00 e9 35 fe ff ff 0f 0b <0f> 0b e9 2c fe ff ff e8 30 b2 5c db e9 22 fe ff ff 0f 0b 0f 0b e9
Sep 02 21:38:44 archbook kernel: RSP: 0018:ffffb4b7483d3dc0 EFLAGS: 00010246
Sep 02 21:38:44 archbook kernel: RAX: 0000000000000000 RBX: ffff982983ee4800 RCX: 0000000000000000
Sep 02 21:38:44 archbook kernel: RDX: 0000000000000000 RSI: 00000000fffffe00 RDI: ffff982983ee4800
Sep 02 21:38:44 archbook kernel: RBP: ffffb4b7483d3e10 R08: 0000000000000000 R09: 0000000000000000
Sep 02 21:38:44 archbook kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffff9828b6811a18
Sep 02 21:38:44 archbook kernel: R13: ffff9829859f3000 R14: ffffb4b7483d3dc0 R15: dead000000000100
Sep 02 21:38:44 archbook kernel: FS:  0000000000000000(0000) GS:ffff982caea00000(0000) knlGS:0000000000000000
Sep 02 21:38:44 archbook kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 02 21:38:44 archbook kernel: CR2: 00007fa88fac7ac8 CR3: 0000000051810002 CR4: 00000000001706f0
Sep 02 21:38:44 archbook kernel: Call Trace:
Sep 02 21:38:44 archbook kernel:  <TASK>
Sep 02 21:38:44 archbook kernel:  ? cfg80211_process_wdev_events+0x15c/0x1b0 [cfg80211 bc1bd9d1424fd243be918a75d325d17bf4ce997c]
Sep 02 21:38:44 archbook kernel:  cfg80211_process_wdev_events+0x15c/0x1b0 [cfg80211 bc1bd9d1424fd243be918a75d325d17bf4ce997c]
Sep 02 21:38:44 archbook kernel:  cfg80211_process_rdev_events+0x2a/0x40 [cfg80211 bc1bd9d1424fd243be918a75d325d17bf4ce997c]
Sep 02 21:38:44 archbook kernel:  cfg80211_event_work+0x29/0x40 [cfg80211 bc1bd9d1424fd243be918a75d325d17bf4ce997c]
Sep 02 21:38:44 archbook kernel:  process_one_work+0x1c4/0x380
Sep 02 21:38:44 archbook kernel:  worker_thread+0x51/0x390
Sep 02 21:38:44 archbook kernel:  ? rescuer_thread+0x3b0/0x3b0
Sep 02 21:38:44 archbook kernel:  kthread+0xdb/0x110
Sep 02 21:38:44 archbook kernel:  ? kthread_complete_and_exit+0x20/0x20
Sep 02 21:38:44 archbook kernel:  ret_from_fork+0x1f/0x30
Sep 02 21:38:44 archbook kernel:  </TASK>
Sep 02 21:38:44 archbook kernel: ---[ end trace 0000000000000000 ]---

Then one times

Sep 04 16:47:55 archbook kernel: ------------[ cut here ]------------
Sep 04 16:47:55 archbook kernel: WARNING: CPU: 3 PID: 56218 at fs/ext4/inode.c:3626 ext4_dirty_folio+0x2c/0x40 [ext4]
Sep 04 16:47:55 archbook kernel: Modules linked in: usbhid xfs btrfs blake2b_generic xor raid6_pq libcrc32c ntfs3 exfat uas usb_storage vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock tun xt_tcpudp rfcomm ip6table_filter ip6_tables iptable_filter cmac algif_hash algif_skcipher af_alg bnep btusb btrtl btbcm btintel btmtk bluetooth ecdh_generic snd_hda_codec_hdmi snd_ctl_led snd_hda_codec_idt mousedev snd_hda_codec_generic ledtrig_audio intel_rapl_msr snd_hda_intel intel_rapl_common vmnet(OE) snd_intel_dspcfg snd_intel_sdw_acpi x86_pkg_temp_thermal snd_hda_codec intel_powerclamp snd_hda_core i915 iTCO_wdt hp_wmi sparse_keymap coretemp intel_pmc_bxt snd_hwdep drm_buddy wl(POE) mei_hdcp mei_pxp iTCO_vendor_support platform_profile wmi_bmof rapl joydev snd_pcm ttm mei_wdt intel_cstate drm_display_helper snd_timer intel_uncore mei_me cfg80211 snd cec psmouse lpc_ich rfkill soundcore e1000e intel_gtt mei wmi hp_accel wireless_hotkey mac_hid video lis3lv02d vmmon(OE) vmw_vmci
Sep 04 16:47:55 archbook kernel:  crypto_user fuse bpf_preload ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 dm_crypt cbc encrypted_keys trusted asn1_encoder tee tpm rng_core dm_mod crct10dif_pclmul crc32_pclmul serio_raw crc32c_intel ghash_clmulni_intel atkbd libps2 vivaldi_fmap aesni_intel crypto_simd cryptd sdhci_pci cqhci sdhci xhci_pci mmc_core xhci_pci_renesas i8042 serio vfat fat
Sep 04 16:47:55 archbook kernel: Unloaded tainted modules: hv_vmbus():1 acpi_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 pcc_cpufreq():1 acpi_cpufreq():1 acpi_cpufreq():1 asus_ec_sensors():1 pcc_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 asus_ec_sensors():1 fjes():1 pcc_cpufreq():1 acpi_cpufreq():1 fjes():1 acpi_cpufreq():1
Sep 04 16:47:55 archbook kernel: CPU: 3 PID: 56218 Comm: vmx-vcpu-2 Tainted: P        W IOE     5.19.6-arch1-1 #1 1307568ef808cb599ee891e507344feaf47e0ed7
Sep 04 16:47:55 archbook kernel: Hardware name: Hewlett-Packard HP EliteBook 8470p/179B, BIOS 68ICF Ver. F.68 07/13/2017
Sep 04 16:47:55 archbook kernel: RIP: 0010:ext4_dirty_folio+0x2c/0x40 [ext4]
Sep 04 16:47:55 archbook kernel: Code: 1e fa 0f 1f 44 00 00 48 8b 06 a8 01 75 07 48 8b 06 a8 08 74 0c 48 83 7e 28 00 74 0e e9 ad e5 cf db 0f 0b 48 83 7e 28 00 75 f2 <0f> 0b e9 9d e5 cf db 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f
Sep 04 16:47:55 archbook kernel: RSP: 0018:ffffb4b7423f7cc0 EFLAGS: 00010246
Sep 04 16:47:55 archbook kernel: RAX: 01ffff0000000077 RBX: fffff58f00cd48c0 RCX: 0000000000000002
Sep 04 16:47:55 archbook kernel: RDX: 0000000000000000 RSI: fffff58f00cd48c0 RDI: ffff982c1ead0c00
Sep 04 16:47:55 archbook kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000400035
Sep 04 16:47:55 archbook kernel: R10: ffff982b85a51c50 R11: 0000000000000001 R12: 0000000000000041
Sep 04 16:47:55 archbook kernel: R13: 0000000000000001 R14: ffff9829a6030c60 R15: 00000000ac0d993e
Sep 04 16:47:55 archbook kernel: FS:  00007f596a5fc6c0(0000) GS:ffff982caeac0000(0000) knlGS:0000000000000000
Sep 04 16:47:55 archbook kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 04 16:47:55 archbook kernel: CR2: 00007f5a10f95000 CR3: 000000036ba24005 CR4: 00000000001706e0
Sep 04 16:47:55 archbook kernel: Call Trace:
Sep 04 16:47:55 archbook kernel:  <TASK>
Sep 04 16:47:55 archbook kernel:  set_page_dirty_lock+0x32/0x80
Sep 04 16:47:55 archbook kernel:  qp_release_pages+0x65/0xf0 [vmw_vmci 6b0f9534fbc69ffd840488667655c626b31c590d]
Sep 04 16:47:55 archbook kernel:  qp_host_unregister_user_memory+0x23/0x80 [vmw_vmci 6b0f9534fbc69ffd840488667655c626b31c590d]
Sep 04 16:47:55 archbook kernel:  vmci_qp_broker_detach+0x19e/0x400 [vmw_vmci 6b0f9534fbc69ffd840488667655c626b31c590d]
Sep 04 16:47:55 archbook kernel:  vmci_host_unlocked_ioctl+0xb1/0xc60 [vmw_vmci 6b0f9534fbc69ffd840488667655c626b31c590d]
Sep 04 16:47:55 archbook kernel:  ? __wake_up_common+0x73/0x180
Sep 04 16:47:55 archbook kernel:  __x64_sys_ioctl+0x91/0xd0
Sep 04 16:47:55 archbook kernel:  do_syscall_64+0x5c/0x90
Sep 04 16:47:55 archbook kernel:  ? do_syscall_64+0x6b/0x90
Sep 04 16:47:55 archbook kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Sep 04 16:47:55 archbook kernel:  ? do_syscall_64+0x6b/0x90
Sep 04 16:47:55 archbook kernel:  ? do_syscall_64+0x6b/0x90
Sep 04 16:47:55 archbook kernel:  ? exit_to_user_mode_prepare+0x16f/0x1d0
Sep 04 16:47:55 archbook kernel:  ? syscall_exit_to_user_mode+0x1b/0x40
Sep 04 16:47:55 archbook kernel:  ? do_syscall_64+0x6b/0x90
Sep 04 16:47:55 archbook kernel:  entry_SYSCALL_64_after_hwframe+0x63/0xcd
Sep 04 16:47:55 archbook kernel: RIP: 0033:0x7f5a7d82d9ef
Sep 04 16:47:55 archbook kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Sep 04 16:47:55 archbook kernel: RSP: 002b:00007f596a5f0630 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Sep 04 16:47:55 archbook kernel: RAX: ffffffffffffffda RBX: 00007f596a5f06a0 RCX: 00007f5a7d82d9ef
Sep 04 16:47:55 archbook kernel: RDX: 00007f596a5f06a0 RSI: 00000000000007aa RDI: 0000000000000077
Sep 04 16:47:55 archbook kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Sep 04 16:47:55 archbook kernel: R10: 000056012face900 R11: 0000000000000246 R12: 00000000fffffffe
Sep 04 16:47:55 archbook kernel: R13: 0000000000000020 R14: 0000000000000020 R15: 00007f5a7c78e280
Sep 04 16:47:55 archbook kernel:  </TASK>
Sep 04 16:47:55 archbook kernel: ---[ end trace 0000000000000000 ]---
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 53391 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54151 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54245 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54789 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54792 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54800 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54801 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54802 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54803 does not have buffers attached
Sep 04 16:48:08 archbook kernel: EXT4-fs warning (device dm-0): mpage_prepare_extent_to_map:2638: inode #4724620: comm kworker/u16:7: page 54804 does not have buffers attached

Edit2:
Please post the output of

find /etc/systemd -name \*.service -type l -a -xtype f -printf %f\\n

Last edited by loqs (2022-09-06 01:29:50)

seth · 2022-09-06 06:46:48

% grep -c "ATA bus error" ofHI.txt
1748
% grep -c "CommWake" ofHI.txt
533
% grep -c "I/O error" ofHI.txt
1051

Since most/all of the IO errors seem to be bus errors, I'd start by disabling TLP.

Sep 02 21:38:44 archbook wpa_supplicant[685]: wlo1: Trying to associate with 62:38:e0:c2:38:1b (SSID='Hunter Bidens Laptop' freq=5200 MHz)
Sep 02 21:38:44 archbook wpa_supplicant[685]: wlo1: CTRL-EVENT-SSID-TEMP-DISABLED id=0 ssid="Hunter Bidens Laptop" auth_failures=1 duration=10 reason=WRONG_KEY

1. this is not how this works …
2. More seriously

% grep -Ec 'CONN_FAILED|WRONG_KEY' ofHI.txt
232
% grep -c 'CTRL-EVENT-DISCONNECTED' ofHI.txt
4200

You're jumping between a flurry of APs and there're segments where this is happening in rapid succession
The SSID 'LETS GO BRANDON ' (with a tailing blank…) shows up with numerous MACs and 5GHz frequencies (02:1a:11:<some counter>) and might be the 5GHz version of the 2.4GHz 'NSA BULK DATA COLLECTOR' but its behavior leads to frequent disconnects.

=> stabilize your network connection, 62:38:e0:c2:38:1b ('Hillarys Mailserver'/'Hunter Bidens Laptop') and 62:38:e0:c2:38:1c ('Ace', both 5GHz) seem particularily unreliable; 'LETS GO BRANDON ' might apply some quota and systematically kick everyone through MAC and frequency changes.

Also/alternatively the __cfg80211_connect_result oopsies seem to be tied to vmnetBridge/vmnet-natd responses to AP scans
=> try the behaviour w/o vmware and/or disable https://wiki.archlinux.org/title/Networ … domization

covid19 · 2022-09-24 20:39:28

Sorry late reply, system has been surprisingly a bit more stable, uptime just reached 24 days. I have a hunch that it's somehow possibly maybe related to vmware

~]$ find /etc/systemd -name \*.service -type l -a -xtype f -printf %f\\n
NetworkManager-wait-online.service
dbus-org.freedesktop.nm-dispatcher.service
dbus-org.bluez.service
getty@tty1.service
systemd-timesyncd.service
expressvpn.service
smb.service
NetworkManager.service
tlp.service
cronie.service
vmware-networks.service
vmware-usbarbitrator.service
dbus-org.freedesktop.timesync1.service
bluetooth.service

Hmmm as for the network disconnects, there were a few days where I was screwing around with the routers firmware and internet was on/off quite a lot. I've personally never had it disconnect like crazy, or observed any flakiness regarding wifi. My vpn may drop/reconnect every 4-5 days but other than that I don't see connection failures or anything.
I went ahead and disabled the mac address randomization

cat /etc/NetworkManager/conf.d/wifi_rand_mac.conf
[device]
wifi.scan-rand-mac-address=no

Aug 21 17:12:39 archbook kernel: Linux version 5.8.2-arch1-1

Certainly not running 5.8.2, more like 5.19.6.arch1-1

Last edited by covid19 (2022-09-24 20:51:36)

covid19 · 2022-10-12 07:56:00

Okay, not sure whether to mark this resolved now. Just hit 40 days and everything has been working great. I think the above SSIDs weren't just fluke, I have multiple virtual SSIDs (One for 5ghz only, one for 2.4ghz only, one for the router to automatically do this for you depending on range/signal strength, and an additional 2 for guest networks, isolated from the rest of the computers, one over 2.4ghz and the other over 5ghz)
Network connection has been rock solid, maybe one-two disconnects a week from my VPN.
Really glad to see it's finally being stable, I had been contemplating a reinstall. Would really really love to have ZFS on root, just waiting for a reason lol, I don't like mucking about with something that's currently WORKING

Thanks for all the support, this thread has been most informative. If I see anymore flakiness, I'll kill tlp and then reevaluate. I had been looking at vmware as a potential culprit, maybe all I needed to do was disable that mac address randomization? Time will tell I suppose, going to aim for 100 day uptime

Last edited by covid19 (2022-10-12 07:57:12)

covid19 · 2022-10-26 20:58:55

I don't see any way to change the title of the post, or how to mark it as solved...been looking around, don't see anything, unless its hidden.
Haven't had any further issues with the laptop at all now for the past 55 days. I do remember my root ssd was throwing errors at one point randomly, so I've bought an EVO 870 (replacing EVO 860). Basically just reading up on ZFS now since I'm starting a new disk, seems neat. Much better snapshot solution than what I currently have (Timeshift+rsync), and I'd rather not Btrfs...at least not at this time. ZFS seems much more mature but it's a shame people are ragging over licensing instead of just interest in creating the greatest product, or at least another very viable reliable option alongside other "license friendly" filesystems.. Though I don't think the average desktop user neither understands, cares, or gives the slightest damn about any of that...myself included

Thanks for the troubleshooting steps and help guys, I learned new things here. Valuable information that I saved

seth · 2022-10-26 21:08:49

Mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

covid19 · 2022-10-27 00:27:32

seth wrote:

Mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Thank you, never thought to look there. Not a huge forum guy, appreciated
Thanks to everyone again

covid19 · 2022-11-09 01:59:28

It ended up taking a shit at 66 days
But next reboot I did I landed @ grub rescue.

symbol grub_disk_native_sectors not found

No idea what that means but I booted usb installer and mounted root.
Saw some tips saying to just reinstall kernel, so i did. mounted efi and reinstalled grub for good measure too.

Now when I attempt a normal boot, the rescue says: 'grub_debug_malloc' not found
Luckily I can smash F9 at boot and boot via the usb installer. And I land @ graphical target, but if I reboot and try to boot as I always have, I get the grub_debug_malloc not found.
How can I correct this? Kinda annoying smashing F9 before luks is demanding the key

Doesn't make sense to me, works one way but not the other

cfr · 2022-11-09 03:18:03

I'd really suggest starting a new thread. People may not look at one marked 'solved' unless they are looking for a solution.

ewaller · 2022-11-09 03:29:49

cfr wrote:

I'd really suggest starting a new thread. People may not look at one marked 'solved' unless they are looking for a solution.

No. Let's keep this all in one thread. This does not seem to be a widespread issue and may have a narrow root cause; perhaps hardware. Perhaps broken due to partial updates that occurred in the past.

covid19 · 2022-11-09 05:01:24

Yeah I was torn between opening a new thread vs just adding onto this one; figured since it's the same ..."issue"
Only updates that I've ever really held back are 'linux' , 'linux-headers' , 'grub' -- to avoid needing a reboot until I'm good and ready, grub because custom patch for menu

It was odd though it's never just locked up like that and then next boot throw me into rescue. That damn screen makes my stomach sink

Even weirder is that nothings changed, but I can no longer just let the laptop startup by itself, have to beat luks prompt hitting F9 and then able to boot from usb

Last edited by covid19 (2022-11-09 05:03:34)

cfr · 2022-11-09 06:22:49

ewaller wrote:

No. Let's keep this all in one thread.

Fair enough. But then maybe it would be better to unmark it solved?

cfr · 2022-11-09 06:30:12

covid19 wrote:

Now when I attempt a normal boot, the rescue says: 'grub_debug_malloc' not found

There have been some other reports of this (e.g. here), which is why I thought you were experiencing a different issue, but they may not be relevant if ewaller thinks this is due to the same underlying problem as earlier. (For all I know, that is a very common error and so likely to have disparate causes.)

seth · 2022-11-09 07:32:21

It ended up taking a shit at 66 days sad
But next reboot I did I landed @ grub rescue.
symbol grub_disk_native_sectors not found
No idea what that means but I booted usb installer and mounted root.
Saw some tips saying to just reinstall kernel, so i did. mounted efi and reinstalled grub for good measure too.
Now when I attempt a normal boot, the rescue says: 'grub_debug_malloc' not found

grub because custom patch for menu

I'd try installing the repo grub… or at least rebuilding grub.

covid19 · 2022-11-09 09:41:11

Yeah, all of this is with repo grub, I only apply a super basic patch to fix the ordering of kernels (lts goes to submenu, main always boots 'linux')

And yup, that thread seems like the same thing as what I'm experiencing

Arch Linux

#1 2022-09-02 20:56:43

Indefinite Instability ~2 years in [SOLVED]

#2 2022-09-02 21:13:11

Re: Indefinite Instability ~2 years in [SOLVED]

#3 2022-09-02 21:29:07

Re: Indefinite Instability ~2 years in [SOLVED]

#4 2022-09-02 21:39:20

Re: Indefinite Instability ~2 years in [SOLVED]

#5 2022-09-02 23:00:23

Re: Indefinite Instability ~2 years in [SOLVED]

#6 2022-09-02 23:06:28

Re: Indefinite Instability ~2 years in [SOLVED]

#7 2022-09-02 23:20:07

Re: Indefinite Instability ~2 years in [SOLVED]

#8 2022-09-03 07:23:23

Re: Indefinite Instability ~2 years in [SOLVED]

#9 2022-09-05 21:59:22

Re: Indefinite Instability ~2 years in [SOLVED]

#10 2022-09-05 22:06:58

Re: Indefinite Instability ~2 years in [SOLVED]

#11 2022-09-05 22:33:21

Re: Indefinite Instability ~2 years in [SOLVED]

#12 2022-09-06 06:46:48

Re: Indefinite Instability ~2 years in [SOLVED]

#13 2022-09-24 20:39:28

Re: Indefinite Instability ~2 years in [SOLVED]

#14 2022-10-12 07:56:00

Re: Indefinite Instability ~2 years in [SOLVED]

#15 2022-10-26 20:58:55

Re: Indefinite Instability ~2 years in [SOLVED]

#16 2022-10-26 21:08:49

Re: Indefinite Instability ~2 years in [SOLVED]

#17 2022-10-27 00:27:32

Re: Indefinite Instability ~2 years in [SOLVED]

#18 2022-11-09 01:59:28

Re: Indefinite Instability ~2 years in [SOLVED]

#19 2022-11-09 03:18:03

Re: Indefinite Instability ~2 years in [SOLVED]

#20 2022-11-09 03:29:49

Re: Indefinite Instability ~2 years in [SOLVED]

#21 2022-11-09 05:01:24

Re: Indefinite Instability ~2 years in [SOLVED]

#22 2022-11-09 06:22:49

Re: Indefinite Instability ~2 years in [SOLVED]

#23 2022-11-09 06:30:12

Re: Indefinite Instability ~2 years in [SOLVED]

#24 2022-11-09 07:32:21

Re: Indefinite Instability ~2 years in [SOLVED]

#25 2022-11-09 09:41:11

Re: Indefinite Instability ~2 years in [SOLVED]

Board footer