You are not logged in.

#76 2015-09-24 08:03:52

mich41
Member
Registered: 2012-06-22
Posts: 796

Re: System freezes

You can search your files in /usr/bin, /usr/lib and /usr/lib32 for xacquire and xbegin using objdump -d and grep.  But I just did that on my system and besides false positives (data constants interpreted as instructions) in mupdf, mono and libcrypto, it only found xbegin in /usr/lib/libpthread.a, /usr/lib/libpthread-2.22.so and /usr/lib/libitm.so.1.0.0. Of these, pthread is part of glibc and libitm probably isn't used by anything at all.

Offline

#77 2015-09-25 20:24:10

andychu
Member
Registered: 2015-09-09
Posts: 1

Re: System freezes

I also got a lot of freezes on my new thinkpad t450s mostly after suspending or sometimes randomly so I'd have to hard reboot the laptop.
I seem to have fixed it now, I have had no crashes or freezes for a couple of days even after suspending a lot of times.
What I did was upgrade to the Arch testing repo so I would be on kernel 4.2.1-1 and adding the i915.enable_execlists=0 kernel parameter to my gummiboot Arch entry.
Maybe this will fix your problem and others too?

P.S: Arch is awesome, long live Arch Linux!

Offline

#78 2015-09-27 15:39:12

mich41
Member
Registered: 2012-06-22
Posts: 796

Re: System freezes

It just occured to me that initramfs contains a copy of libpthread.so and if any processes started by initramfs survive to the final system, they will keep using TSX.

Somebody may try patching glibc and re-running mkinitcpio before reboot.

Offline

#79 2015-09-27 16:53:28

kris7t
Member
Registered: 2013-04-25
Posts: 20

Re: System freezes

I ran

find /usr/lib -execdir bash -c "objdump -d '{}' 2>&1 | grep -E 'xacquire|xbegin'" \; -print
find /usr/bin -execdir bash -c "objdump -d '{}' 2>&1 | grep -E 'xacquire|xbegin'" \; -print

to find libaries and binaries using TSX. To my surpruse, the only true positives were libpthread, libitm and linpack_xenon64. Despite being compiled with --enable-lock-elision=no, glibc still has some xbegin instructions in its text, however, calling functions which contain these instructions are disabled by the glibc-2.19-really-disable-hle patch -- which defines HAS_RTM to (0), pretending that cpuinfo does not report support for HLE at runtime --, so in theory, they should cause no problem. I also re-run mkinitcpio to get rid of HLE in early userspace, too. Unfortunately, I am still unable to compile glibc without a kernel panic when SpeedStep is enabled.

Offline

#80 2015-09-27 18:29:04

mich41
Member
Registered: 2012-06-22
Posts: 796

Re: System freezes

Crap. I also can't imagine this patch not doing what it's supposed to do. Pray to Intel and MSI for a BIOS update, then smile

Offline

#81 2015-09-29 12:24:44

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

I have been using Fedora 22 (my god, dnf is terrrrrible) for a several days and it is indeed stable (with various kernels) with every Intel option enabled in BIOS. Then I did 'system-upgrade' to 23 and it hanged right after reboot, then several times more. I rolled back to Fedora 22 and did various tests with kernel and glibc from fc23, results are:

kernel-(4.0.4-4.1.7)-fc22 + glibc-2.21-fc22 = OK
kernel-4.2.1-fc23 + glibc-2.21-fc22 = OK
kernel-4.2.1-fc23 + glibc-2.22-fc23 = FREEZE
kernel-4.0.4-fc22 + glibc-2.22-fc23 = FREEZE

So glibc is definitely the culprit. Either our glibc-2.21 or glibc-2.21 with fedora's patches should make Arch stable.
Can somebody roll back to glibc-2.21 and test this? (dont forget to mkinitcpio) I would try this myself but I need this machine stable for a work urgently.

Last edited by Goresome (2015-09-29 12:26:16)

Offline

#82 2015-09-29 12:41:04

spirtbrat
Member
Registered: 2011-02-19
Posts: 16

Re: System freezes

It's hard testing the proposed solutions since the freezes are so random. I'm experiencing system freezes from anything between 5 minutes and 3 days after a boot. It never, ever freezes when left idle. Can't leave it there and later find the machine frozen. The freeze frequency also does not appear to be load sensitive, e.g. I can't trigger a freeze quicker if I stress the machine.
Does anyone have a (semi-)reliable way to trigger these system freezes?

Offline

#83 2015-09-29 13:26:05

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

spirtbrat wrote:

It's hard testing the proposed solutions since the freezes are so random. I'm experiencing system freezes from anything between 5 minutes and 3 days after a boot. It never, ever freezes when left idle. Can't leave it there and later find the machine frozen. The freeze frequency also does not appear to be load sensitive, e.g. I can't trigger a freeze quicker if I stress the machine.
Does anyone have a (semi-)reliable way to trigger these system freezes?

glibc compilation.

Offline

#84 2015-09-29 13:32:15

spirtbrat
Member
Registered: 2011-02-19
Posts: 16

Re: System freezes

Unable to freeze it that way.
However, there are 4 libpthread segfaults registered in journald during the compilation.

Offline

#85 2015-09-29 13:35:08

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

spirtbrat wrote:

Unable to freeze it that way.
However, there are 4 libpthread segfaults registered in journald during the compilation.

Any kind of gcc usage (like even during 'configure') causes MCE with my 5675C. As for pthread segfaults you need to recompile glibc with --enable-elision-lock=no to disable bugged Hardware Lock Elision.

Offline

#86 2015-09-30 05:32:39

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

Hey - apparently there's a microcode update out to fix this at least to manufacturers. My MSI PX60-2QD got a UEFI update today with the note "updates microcode to 13" that fixes it for me at least when I tried installing an affected OS in a Virtualbox VM (which crashed it before in the same way as running it ). I have no idea if other manufacturers have put one out yet, but you can at least be on the lookout for an Intel fix to this.
http://www.msi.com/support/nb/PX60-2QD.html#down-bios

Interestingly, the MSI forum notes that this same issue was hitting people on Windows with certain apps... Includeing DOTA 2 and OneNote. Did they not test these CPUs at ALL?
https://forum-en.msi.com/index.php?topi … msg1498718

I'll try installing Arch on Thursday (I don't want to break anything until then possibly, I've got some important needs for this laptop until then), which will check the full case.

EDIT: Someone on the kernel bug tracker for this extracted the microcode update for 5700hq CPUs like the MSI update, if you want to try loading it at boot time: https://bugzilla.kernel.org/show_bug.cgi?id=103351 http://benjam.info/downloads/5700hq-ucode.tar.gz

Last edited by GourdCaptain (2015-09-30 05:54:59)

Offline

#87 2015-09-30 05:57:30

AdrianCohea
Member
Registered: 2013-06-29
Posts: 16

Re: System freezes

On the Core i7-5700HQ with 4.2.1-1-ARCH, I can reproduce the issue 100% of the time (5 trials) by doing the following:

yaourt -S metasploit

The kernel panic is always the same: "not syncing - not all CPUs entered the broadcast exception handler"

On reboot, I check what's going on with mcelog:

journalctl -b 0 --unit=mcelog

and I always find something saying "MCA Internal Timer Error" despite no problems with heat whatsoever. There's a kernel bug, and apparently MSI has BIOS update/microcode for this...


gpg --keyserver hkps.pool.sks-keyservers.net --recv-key 0x77660e16dba99933

0B85 1AC2 4980 B654 0DA3  D1E1 7766 0E16 DBA9 9933

Offline

#88 2015-09-30 05:59:03

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

AdrianCohea wrote:

On the Core i7-5700HQ with 4.2.1-1-ARCH, I can reproduce the issue 100% of the time (5 trials) by doing the following:

yaourt -S metasploit

The kernel panic is always the same: "not syncing - not all CPUs entered the broadcast exception handler"

On reboot, I check what's going on with mcelog:

journalctl -b 0 --unit=mcelog

and I always find something saying "MCA Internal Timer Error" despite no problems with heat whatsoever. There's a kernel bug, and apparently MSI has BIOS update/microcode for this...

Try the extracted microcode - http://benjam.info/downloads/5700hq-ucode.tar.gz , you can apparently install them on each boot with iucode-tool.

Offline

#89 2015-09-30 08:51:56

mich41
Member
Registered: 2012-06-22
Posts: 796

Re: System freezes

If you read the whole thread, you will see that there are few known hardware bugs in these CPUs for which Intel claims that "BIOS workarounds are possible".

There were also two people with the exact same MSI notebook and CPU model and only one of them had problems.

Maybe the BIOS fixes are already released by most manufacturers while you are just sitting here for some ucode update or software workaround which will never come?

So far there is not a single word here from anybody who would have found out that there are no BIOS updates available for his machine or that updating doesn't help.

Offline

#90 2015-09-30 08:56:16

kris7t
Member
Registered: 2013-04-25
Posts: 20

Re: System freezes

I was running the latest MSI BIOS (10E) for my machine and had crashes with SpeedStep enabled, and I suppose the other guys here/on the kernel bug tracker with MSI laptops were doing the same. Today (yesterday, depending on time zone...), BIOS 110 was released. It's only changelog entry says it contains the intel ucode update 13. Now my machine does not crash, even with unpatched glibc. So I think it's pretty clear that "BIOS workarounds are possible" == "BIOS may contain updated ucode" here. However, we will see whether ucode extracted from out BIOS update will work on other machines or not...

EDIT: Benjamin Woodruff has released a script to apply the microcode update on non-MSI motherboards: https://github.com/bgw/bdw-ucode-update-tool Apparently, it fixed issues he was having with his i5-5675C!

Last edited by kris7t (2015-10-01 06:33:29)

Offline

#91 2015-10-01 18:51:50

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

The idea with glibc2.21 looked well on paper (I even eliminated panics during compilations), but Arch package contains static libs, so lot of things still contain 2.22 code.

Apparently, Intel has got around fixing their shit and motherboard vendors now have microcode update for our CPUs. Some good guy extracted microcode blobs from MSI BIOS https://bugzilla.kernel.org/show_bug.cgi?id=103351#c28
I can confirm that 0x12 microcode fixes MCE and glibc HLE bug aswell.

Offline

#92 2015-10-01 21:46:56

jonessen
Member
Registered: 2013-12-21
Posts: 27

Re: System freezes

That's a bit of an overstatement, it seems there's only one motherboard vendor (MSI) who has released an update, and Intel doesn't distribute the microcode directly either.
But there is a fix, seemingly for all CPUs with this problem, that seems to work for everyone so far (I have confirmed it for my i7-5775C).

Offline

#93 2015-10-01 21:48:10

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

Yeah. I've got an affected MSI laptop, installed the BIOS update, installed Arch today, and so far everything's going well. smile

Intel has been rather shameful about this, especially given it screws up Windows systems as well...

Offline

#94 2015-10-01 22:37:58

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

jonessen wrote:

That's a bit of an overstatement, it seems there's only one motherboard vendor (MSI) who has released an update, and Intel doesn't distribute the microcode directly either.
But there is a fix, seemingly for all CPUs with this problem, that seems to work for everyone so far (I have confirmed it for my i7-5775C).

Yeah, I'm also enraged by such user treatment from Intel. What I meant was motherboard vendors were given an update (probably all of them, it takes up to half a year to release new BIOS update), but users were given nothing.

Offline

#95 2015-10-02 06:18:40

xazax
Member
Registered: 2015-09-23
Posts: 6

Re: System freezes

I did not have any kernel panics since I updated the bios. Out of curiosity, does intel release any kind of changelogs? I wonder what instructions were problematic and whether they really fixed it or just disabled them.

Offline

#96 2015-10-02 17:26:23

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

xazax wrote:

I did not have any kernel panics since I updated the bios. Out of curiosity, does intel release any kind of changelogs? I wonder what instructions were problematic and whether they really fixed it or just disabled them.

From what I understand, not even for the ones they officially release for use outside of BIOS updates.

Offline

#97 2015-10-06 22:44:00

Pythux
Member
Registered: 2015-10-06
Posts: 1

Re: System freezes

Having the same issue with my 5700hq CPU, the microcode update solves the kernel panic problem. I'm wondering about the best way to apply the change at boot time. Maybe using iucode-tool? I'm not sure about how to do it properly.

Thanks for the help so far.

Offline

#98 2015-10-06 22:47:44

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

Pythux wrote:

Having the same issue with my 5700hq CPU, the microcode update solves the kernel panic problem. I'm wondering about the best way to apply the change at boot time. Maybe using iucode-tool? I'm not sure about how to do it properly.

Thanks for the help so far.

What'd probably be the best option is to patch them into the GenuineIntel.bin file in the /boot/intel-ucode.img initrd provided by intel-ucode, that's a set of microcode files applied automatically at boot. I'm just not sure how you add extra microcodes to that file.

Offline

#99 2015-10-08 10:12:02

Goresome
Member
From: Russia
Registered: 2011-12-06
Posts: 67

Re: System freezes

Everyone using that microcode update should know that HLE crashes still happen, unless you rebuild glibc without elision lock https://bugzilla.kernel.org/show_bug.cgi?id=103351#c45

Offline

#100 2015-10-08 16:52:36

GourdCaptain
Member
Registered: 2009-04-18
Posts: 121

Re: System freezes

Goresome wrote:

Everyone using that microcode update should know that HLE crashes still happen, unless you rebuild glibc without elision lock https://bugzilla.kernel.org/show_bug.cgi?id=103351#c45

As one of the people with the MSI update, TSX and such are disabled on my system according to checking /proc/cpuinfo. I think applying it during early userspace would also have that effect?

Last edited by GourdCaptain (2015-10-08 16:52:59)

Offline

Board footer

Powered by FluxBB