Arch Linux

ttheodor

Member

Registered: 2013-12-29

Posts: 31

[solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

Hi,

two days ago I updated my X1 Nano's (Tiger Lake) packages (via pacman -Syu) and it became unusable. The last update was sometime in November.

A few of the symptoms (some of them, e.g., the wifi one, don't occur with every boot):

- The wifi adapter is not available, i.e., NetworkManager doesn't "see" it.

- The status bar I'm using (i3-bar) doesn't not appear.

- GUI programs (e.g., Firefox) won't start

- sudo hangs: I don't even see a password prompt

- su prints: free(): invalid pointer

- shutting down/rebooting waits indefinitely for services to stop

I plugged a live USB stick and arch-chroot'd to downgrade everything by setting /etc/pacman.d/mirrorlist to Server=https://archive.archlinux.org/repos/2021/11/01/$repo/os/$arch.
Initially this worked, but then I tried searching for a more recent working date to identify which package is causing this but I couldn't find the culprit.
Moreover, downgrading back to 2021/11/01 no longer resolves the issue, could it be that pacman changes something while installing an update that is not being undone when downgrading?

Using a live USB, Arch Linux or Windows 10, works fine.

Any ideas on how I can proceed with debugging/figuring this out? Thanks in advance!

Last edited by ttheodor (2021-12-22 14:53:51)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,081

Re: [solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

Post the full output of a

journalctl -b

ran as root: https://wiki.archlinux.org/title/List_o … n_services as well as your /var/log/pacman.log covering at least the breaking period.

That kind of failure, assuming no package inconsistency, might more generally point to RAM or hard disk issues, I suggest you run a SMART test and post the smartctl -a output of the relevant drive, and maybe run a memtest for a few passes overnight.

ttheodor

Member

Registered: 2013-12-29

Posts: 31

Re: [solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

journal (that's with the downgraded system which is still in a non-working state): http://ix.io/3J8a

pacman.log  for the breaking update: http://ix.io/3J8d

Last edited by ttheodor (2021-12-22 13:41:51)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,081

Re: [solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

Something we immediately see here is that you have at least three networking services enabled, which all get into a conflict and fight over the network controller (which is the reason for at least the first issue in your post) and can lead to all sorts of weird stuff. If it's your intention to use NetworkManager disable and stop systemd-networkd, systemd-resolved and iwd potentially other stuff you have related to networking/maybe post

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

the only thing network related in that list should be NetworkManager.

This can lead to all sorts of weird shit, so this might possibly be the fix already, also disable TLP to make sure there's no conflict with some overzealous power saving setting.

In general the journal from a "broken" boot might give more additional information you can do -b-1 for the previous boot -2 for the one before that and so forth, so we can get a log from an actually broken boot, maybe also the actual downgrade from the pacman.log since this does not cover that and you appear to at least run a kernel currently that is not covered by this excerpt.

Last edited by V1del (2021-12-22 14:32:44)

ttheodor

Member

Registered: 2013-12-29

Posts: 31

Re: [solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

disable and stop systemd-networkd, systemd-resolved and iwd ... also disable TLP

That was it! Thanks so much!

For future reference: how did you figure this out? Was there something obvious in the logs?

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
bluetooth.service                        | bluetooth.target.wants
dbus-org.bluez.service                   | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.timesync1.service   | system
dirmngr.socket                           | sockets.target.wants
display-manager.service                  | system
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gpg-agent-browser.socket                 | sockets.target.wants
gpg-agent-extra.socket                   | sockets.target.wants
gpg-agent.socket                         | sockets.target.wants
gpg-agent-ssh.socket                     | sockets.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire.socket                          | sockets.target.wants
pulseaudio.socket                        | sockets.target.wants
remote-fs.target                         | multi-user.target.wants
systemd-timesyncd.service                | sysinit.target.wants 

P.S.: Where is the "buy me a beer" button?

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 25,081

Re: [solved]Thinkpad X1 Nano unusable after upgrade,downgrade doesn't help

Well I listed these four specifically because they all appear in the log in some form, we even have a cool crash of the wifi subsystem in the form of

Dec 22 14:29:38 dogfish kernel: CPU: 3 PID: 1146 Comm: iwd Tainted: G           OE     5.14.15-arch1-1 #1 00839a97657f819e080ee52ba6354f3abf438ad2
Dec 22 14:29:38 dogfish kernel: Hardware name: LENOVO 20UQS04L00/20UQS04L00, BIOS N2TET69W (1.47 ) 10/19/2021
Dec 22 14:29:38 dogfish kernel: RIP: 0010:nl80211_get_reg_do+0x23c/0x2b0 [cfg80211]
Dec 22 14:29:38 dogfish kernel: Code: 0c 01 00 00 00 e8 84 0a eb f4 85 c0 74 cc e9 f5 fe ff ff 48 89 ef 48 89 04 24 e8 4f e4 1e f5 e8 aa 9d 21 f5 48 8b 04 24 eb 84 <0f> 0b 48 89 ef e8 3a e4 1e f5 e8 95 9d 21 f5 b8 ea ff ff ff e9 6b
Dec 22 14:29:38 dogfish kernel: RSP: 0018:ffffb56040c6ba60 EFLAGS: 00010202
Dec 22 14:29:38 dogfish kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
Dec 22 14:29:38 dogfish kernel: RDX: ffff9e3eff9d8008 RSI: 0000000000000000 RDI: ffff9e3eff9d82e0
Dec 22 14:29:38 dogfish kernel: RBP: ffff9e3f0688ae00 R08: 0000000000000004 R09: ffff9e3f01196014
Dec 22 14:29:38 dogfish kernel: R10: 0000000000000021 R11: ffff9e3ec5d5cc00 R12: ffffb56040c6bac0
Dec 22 14:29:38 dogfish kernel: R13: ffff9e3f01196014 R14: 0000000000000000 R15: ffff9e3eff9d82e0
Dec 22 14:29:38 dogfish kernel: FS:  00007fd55211e740(0000) GS:ffff9e420f6c0000(0000) knlGS:0000000000000000
Dec 22 14:29:38 dogfish kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 22 14:29:38 dogfish kernel: CR2: 00007ffe6b3e7f20 CR3: 0000000145694003 CR4: 0000000000770ee0
Dec 22 14:29:38 dogfish kernel: PKRU: 55555554
Dec 22 14:29:38 dogfish kernel: Call Trace:
Dec 22 14:29:38 dogfish kernel:  genl_family_rcv_msg_doit+0xfa/0x160
Dec 22 14:29:38 dogfish kernel:  genl_rcv_msg+0xeb/0x1e0
Dec 22 14:29:38 dogfish kernel:  ? __cfg80211_rdev_from_attrs+0x1d0/0x1d0 [cfg80211 4ec19b6568393f10b5f0323c03e409adefc11da4]
Dec 22 14:29:38 dogfish kernel:  ? nl80211_send_regdom.constprop.0+0x1b0/0x1b0 [cfg80211 4ec19b6568393f10b5f0323c03e409adefc11da4]
Dec 22 14:29:38 dogfish kernel:  ? genl_get_cmd+0xd0/0xd0
Dec 22 14:29:38 dogfish kernel:  netlink_rcv_skb+0x59/0x100
Dec 22 14:29:38 dogfish kernel:  genl_rcv+0x24/0x40
Dec 22 14:29:38 dogfish kernel:  netlink_unicast+0x23b/0x350
Dec 22 14:29:38 dogfish kernel:  netlink_sendmsg+0x23d/0x480
Dec 22 14:29:38 dogfish kernel:  sock_sendmsg+0x5b/0x60
Dec 22 14:29:38 dogfish kernel:  __sys_sendto+0x124/0x190
Dec 22 14:29:38 dogfish kernel:  __x64_sys_sendto+0x20/0x30
Dec 22 14:29:38 dogfish kernel:  do_syscall_64+0x59/0x80
Dec 22 14:29:38 dogfish kernel:  ? do_syscall_64+0x69/0x80
Dec 22 14:29:38 dogfish kernel:  ? __audit_syscall_exit+0x24d/0x2a0
Dec 22 14:29:38 dogfish kernel:  ? syscall_exit_to_user_mode+0x23/0x40
Dec 22 14:29:38 dogfish kernel:  ? do_syscall_64+0x69/0x80
Dec 22 14:29:38 dogfish kernel:  ? do_syscall_64+0x69/0x80
Dec 22 14:29:38 dogfish kernel:  ? exc_page_fault+0x72/0x170
Dec 22 14:29:38 dogfish kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Dec 22 14:29:38 dogfish kernel: RIP: 0033:0x7fd552220c10
Dec 22 14:29:38 dogfish kernel: Code: c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 1d 45 31 c9 45 31 c0 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 68 c3 0f 1f 80 00 00 00 00 55 48 83 ec 20 48
Dec 22 14:29:38 dogfish kernel: RSP: 002b:00007ffe6b3ecbf8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 22 14:29:38 dogfish kernel: RAX: ffffffffffffffda RBX: 0000563e280cc870 RCX: 00007fd552220c10
Dec 22 14:29:38 dogfish kernel: RDX: 000000000000001c RSI: 0000563e280d84f0 RDI: 0000000000000004
Dec 22 14:29:38 dogfish kernel: RBP: 0000563e280e1e50 R08: 0000000000000000 R09: 0000000000000000
Dec 22 14:29:38 dogfish kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe6b3ecc60
Dec 22 14:29:38 dogfish kernel: R13: 00007ffe6b3ecc5c R14: 0000563e280d6500 R15: 0000563e2801f3fd
Dec 22 14:29:38 dogfish kernel: ---[ end trace 5ab5711a69f802a1 ]---

The only network related things you should see is NetworkManager and wpa_supplicant and the last one should be first mentioned in the form of a

Dec 22 14:29:38 dogfish dbus-daemon[1140]: [system] Activating via systemd: service name='fi.w1.wpa_supplicant1' unit='wpa_supplicant.service' requested by ':1.5' (uid=0 pid=1145 comm="/usr/bin/NetworkManager --no-daemon ")

to show that it's being invoked by NetworkManager and not through other means.

FWIW I'd say you can check whether you can enable TLP again, because while there "might" be issues depending on general HW support for the power saving knobs it tries to set, having conflicting services is definitely a major problem you need to get rid of first.