Lockups Before I Do Anything

theherk · 2014-11-06 18:55:42

I have a laptop, Asus X551MAV with Samsung EVO 840 disk, running Arch for about 2 months now. A few days back I ran a system update -Syu. I had to leave while it was running. On down the road a few days, I finally get back to what I was doing. When I open up the computer, I get a few characters into my password in LightDM and it locks. Can't switch TTY or do anything.

I boot live, chroot in, and disable the display manager. Then when I restart, immediately after the login prompt, before I can type anything I start getting trace data. No errors anywhere. The last few things in the list are about the video and DRM init, but no errors. If I wait, the trace continues at about 135 seconds. The only errors I can see in this section indicate a lockup detected in CPU 1 or 0. Then it says end trace. Waiting for several minutes yields no further details.

I did another arch-chroot and ran mkinitcpio, but the result is the same.

Forgive me, I don't know where to go from here, and I have not been successful in my research. What is the right troubleshooting from here?

edit 1: I should add that I have looked through the logs for something that stuck out, but don't know what to report. The upgrade did update the kernel to 3.17.1-1-ARCH. I can see this in the pacman.log. Should I roll back? Otherwise, it seems to have completed successfully.

edit 2: Downgrading to 3.16 did resolve the issue. Currently re-upgrading to ensure the issue returns. If it does, how do I chase down the cause / or ever upgrade the kernel?

Last edited by theherk (2014-11-06 19:55:00)

MoonSwan · 2014-11-06 19:56:32

If I were in your shoes I would likely roll back the kernel. Failing that I'd look at the pacman logs and see which apps were the last ones to be updated a few days ago then try to discern which is the likeliest culprit and roll that back too. I'm not saying finding the right apps to roll back is an easy task but it is the only way I can think of to undo this cockup. Please post back with success, failure and/or errors.

Edit 1 too:

If it does, how do I chase down the cause / or ever upgrade the kernel?

For now, you don't. It may be best to put the currently-used kernel into the pacman.conf file as a package not to be upgraded. Wait till the next kernel release, upgrade and then see what happens. Re-upgrading your apps one at a time may also be necessary to narrow down which one is giving you grief.

Last edited by MoonSwan (2014-11-06 20:02:29)

theherk · 2014-11-06 20:00:58

Thanks MoonSwan. I did roll back the kernel to 3.16, and the problem resolved. I then upgraded to 3.17 again to verify it returned. It did. So, I am going to roll back again. What troubles me is whether I will ever be able to upgrade; what is causing this I wonder.

MoonSwan · 2014-11-06 20:05:00

Let's figure out what is causing this issue and then we can worry about the why.

jasonwryan · 2014-11-06 20:13:10

What is in your journal?

theherk · 2014-11-06 20:13:27

I guess, that is really what I am trying to do. I have discovered that kernel 3.17.1-1-ARCH is the issue, and I have repeated steps to verify this. So, in a few moments I will have a working system again at 3.16. Disaster averted. Now I am left wondering what causes 3.17 this issue on my machine; the why you mention. Is it worth digging deeper, and if so, how do I do that? If not, I must sort out how to lock the kernel so future system updates don't upgrade the kernel.

jasonwryan · 2014-11-06 20:15:53

See https://bugs.archlinux.org/task/42353

theherk · 2014-11-06 20:16:30

jasonwryan, how do I come upon this information? Right now, since I have not rolled back to 3.16 again yet, I cannot do anything unless in a chroot environment. Shall I chroot in and get the journal output?

Also, since upgrading to 3.17 again it is now not stopping the trace at 135 seconds. I am getting trace data again at 440 seconds. In this trace, it tells me that the journal service is repeatedly stopping and starting.

jasonwryan · 2014-11-06 20:20:34

It doesn't matter which kernel you boot with, you can recover the journal for previous boots (see `man journalctl`).

MoonSwan · 2014-11-06 20:28:10

It seems that Mr. JWR is, as usual, on top of things and has answered my own question re: why this is happening. Thank you for stepping in, Sir.

Last edited by MoonSwan (2014-11-06 20:28:38)

jasonwryan · 2014-11-06 20:31:14

Well, I'm not convinced it is the same issue, but the OP can use the info in that bug report to chase their issue down a little more...

MoonSwan · 2014-11-06 20:33:19

Point taken. I am wondering how the OP is doing too. Getting kind of worried because he hasn't posted back since 20 mins ago.

theherk · 2014-11-06 20:44:04

Sorry guys. I'm having a silly hangup. I didn't know about the -b switch for journalctl (awesome). I am looking at the output and not seeing anything interesting. Nevertheless, I want to provide them to you, but my hangup is not knowing how to do so.

So stupid, but I have the dumps on that machine for the past few boots and don't know how to get them to this machine. I'm looking for another usb drive, so I can move the files.

MoonSwan · 2014-11-06 20:51:39

This is where ssh is your friend. SSH into the problematic machine, copy the logs to the machine you're using to SSH in and problem solved. I realise this isn't going to necessarily help you right now but for future reference it may be a good idea to have this set up. It's what I do to copy logs and crap from a problematic computer to my working machine.

theherk · 2014-11-06 21:01:23

Okay, I found a suitable USB drive. So here are the previous 3 boots of journalctl output. I'm not asking anybody to look diligently through it, but I don't know where to look. I am continuing to look though.

shephard - journalctl -b -1
shephard - journalctl -b -2
shephard - journalctl -b -3

theherk · 2014-11-06 21:07:34

Searching for an error similar to that in the bug report, I find:

Nov 06 13:59:07 shephard kernel: BUG: unable to handle kernel paging request at 00000000fed08004
...
Nov 06 13:59:07 shephard kernel: BUG: scheduling while atomic: systemd-udevd/145/0x00000003

However, that is only in the -b -1 output. I would expect it in -b -3 as well since that is also a 3.17 boot.

Last edited by theherk (2014-11-06 21:08:54)

jasonwryan · 2014-11-06 21:13:56

Ugh. pastebin.com.

-b3 shuts down normally. Are you using an intel GPU?

MoonSwan · 2014-11-06 21:16:20

I skimmed through the last one and unfortunately I can't see anything wrong either. The only other log I can think of to look into is the /var/log/errors or messages log files. After that I'm out of suggestions.

theherk · 2014-11-06 21:18:47

Shoot; sorry about the pastebin.com usage. I won't let it happen again. I am using Intel HD built-in.

jasonwryan · 2014-11-06 21:20:46

Apply the patch: it will probably fix you issue. Or install mainline.

theherk · 2014-11-06 21:20:49

Also, -b -1 is probably the best to go off of. -b -3 may have been the boot after the rollback but before I fixed gummiboot, which caused it to still think it booted 3.17 and put me in emergency mode. Therefore, it may be a totally different issue altogether. However, -b -1 is certainly a recreation of the error condition.

theherk · 2014-11-07 03:33:20

Whoa! That was a very long troubleshooting step. That took like 3 hours to compile. At any rate, assuming I have applied the patch correctly, it did not sort my issue. The issue is the same.

I added the patch to my prepare(), and it was copied to my src directory, so I think it patched. I think I'll just stick with 3.16 for a while. Thank you sincerely for all of your help. If there are other steps I should take, let me know. It is not a pressing issue though since I have a workaround.

Arch Linux

#1 2014-11-06 18:55:42

Lockups Before I Do Anything

#2 2014-11-06 19:56:32

Re: Lockups Before I Do Anything

#3 2014-11-06 20:00:58

Re: Lockups Before I Do Anything

#4 2014-11-06 20:05:00

Re: Lockups Before I Do Anything

#5 2014-11-06 20:13:10

Re: Lockups Before I Do Anything

#6 2014-11-06 20:13:27

Re: Lockups Before I Do Anything

#7 2014-11-06 20:15:53

Re: Lockups Before I Do Anything

#8 2014-11-06 20:16:30

Re: Lockups Before I Do Anything

#9 2014-11-06 20:20:34

Re: Lockups Before I Do Anything

#10 2014-11-06 20:28:10

Re: Lockups Before I Do Anything

#11 2014-11-06 20:31:14

Re: Lockups Before I Do Anything

#12 2014-11-06 20:33:19

Re: Lockups Before I Do Anything

#13 2014-11-06 20:44:04

Re: Lockups Before I Do Anything

#14 2014-11-06 20:51:39

Re: Lockups Before I Do Anything

#15 2014-11-06 21:01:23

Re: Lockups Before I Do Anything

#16 2014-11-06 21:07:34

Re: Lockups Before I Do Anything

#17 2014-11-06 21:13:56

Re: Lockups Before I Do Anything

#18 2014-11-06 21:16:20

Re: Lockups Before I Do Anything

#19 2014-11-06 21:18:47

Re: Lockups Before I Do Anything

#20 2014-11-06 21:20:46

Re: Lockups Before I Do Anything

#21 2014-11-06 21:20:49

Re: Lockups Before I Do Anything

#22 2014-11-07 03:33:20

Re: Lockups Before I Do Anything

Board footer