[SOLVED]Random errors while compiling C++ code

GloW_on_dub · 2019-06-06 14:25:44

I have random errors when compiling software. It is not related to the software source code.
Everytime an error occurs, I can just retrigger a build and it will work without issue.

On a 16000 target project, I usually have two or three errors before being able to complete the build.

Here are example of errors I have been having :

cc1plus: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.archlinux.org/> for instructions.

/tmp/cc7XJAoQ.s: Assembler messages:
/tmp/cc7XJAoQ.s:19436: Error: junk at end of line, first unrecognized character valued 0x1a

collect2: fatal error: ld terminated with signal 7 [Bus error], core dumped
compilation terminated.

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

/tmp/ccdvTRIV.s: Assembler messages:
/tmp/ccdvTRIV.s: Internal error (Segmentation fault).
Please report this bug.

I suspected hardware issue, even though I had no other problem. mprime(blend, 2hours) and smartctl (long test) did not find anything.

I do not know where to go from here except reinstalling. If you have an idea I will take it !

Last edited by GloW_on_dub (2019-07-15 09:00:11)

ewaller · 2019-06-06 14:44:00

When you restart, are you doing a clean build? or are you starting from where you failed?
Are you using a build system? Which one?
Do you have swapping configured and enabled? Is the system thrashing? (Repeated page faults and subsequent extreme disk action)
Of no swapping, is the kernel reaching out and killing a process for lack of resources? (Check your journal)

I am thinking either a memory leak or something getting corrupted.
How is the processor doing thermally?

GloW_on_dub · 2019-06-06 15:45:39

> When you restart, are you doing a clean build? or are you starting from where you failed?
I'm starting from where I failed.

>Are you using a build system? Which one?
CMake + Ninja

>Do you have swapping configured and enabled?
Swapping is disabled

> Is the system thrashing? (Repeated page faults and subsequent extreme disk action)
How do I diagnose that ?

> If no swapping, is the kernel reaching out and killing a process for lack of resources? (Check your journal)
I will check as soon as I reproduce the bug again.

> How is the processor doing thermally?
While compiling :
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C (crit = +105.0°C)
temp2: +29.8°C (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +70.0°C (high = +80.0°C, crit = +100.0°C)
Core 0: +70.0°C (high = +80.0°C, crit = +100.0°C)
Core 1: +70.0°C (high = +80.0°C, crit = +100.0°C)
Core 2: +66.0°C (high = +80.0°C, crit = +100.0°C)
Core 3: +65.0°C (high = +80.0°C, crit = +100.0°C)

Last edited by GloW_on_dub (2019-06-06 15:45:50)

GloW_on_dub · 2019-06-06 15:53:17

Nothing appears in journalctl nor in dmesg.

ewaller · 2019-06-06 15:54:26

So, it is not banging its head on the thermal design (noted that those temps were taken during the compile). No throttling should be occurring.
It is not thrashing since swapping is disabled. So, check the kernel to see if it is killing processes due to low memory.
If, after failure, you do a clean build, does it fail again at approximately the same spot?

Alad · 2019-06-06 16:04:42

type c++ g++
g++ --version
g++ -E -Wp,-v - </dev/null

Also check available disk/memory space, e.g if /tmp is full

Last edited by Alad (2019-06-06 16:10:02)

GloW_on_dub · 2019-06-07 07:27:46

[glow@snowman ~/Downloads]$ type c++ g++
c++ is /usr/bin/c++
g++ is /usr/bin/g++
[glow@snowman ~/Downloads]$ g++ --version
g++ (GCC) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[glow@snowman ~/Downloads]$ g++ -E -Wp,-v - </dev/null
ignoring nonexistent directory "/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include
 /usr/local/include
 /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include-fixed
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 31 "<command-line>"
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 32 "<command-line>" 2
# 1 "<stdin>"

tmp is not full, memory is not filled up.

> So, check the kernel to see if it is killing processes due to low memory.
How do I do that ?

> If, after failure, you do a clean build, does it fail again at approximately the same spot?
I just completed a full build without error this time. I will do a few more.

seth · 2019-06-07 13:09:01

Keep an eye on the free memory while compiling. If you're approaching the available limit there's a high chance that you run OOM.
Also setting up reasonably sized swap space (file or partition) should stop this behavior in case.

GloW_on_dub · 2019-06-09 11:21:56

I never run out of memory while compiling and having this error, always around 6Gb free.

crondog · 2019-06-09 21:01:57

Are you sure that it is always that much free? Does it ever spike?

What happens if you built with ninja -j2 or something small like that? Only asking since I have come across projects which I need to limit the number of linker job pools since it uses lots of ram while linking and then crashes my system (eg LLVM in debug)

rowdog · 2019-06-11 13:37:32

Bad RAM? A memory stick that's failing intermittently can produce segfaults. I see that mprime will do a good stress test but I'm not sure it is as thorough at testing RAM as memtest86+

GloW_on_dub · 2019-06-12 10:00:20

I just had the issue while tracing the memory, here is how it looks:
https://i.ibb.co/rMzTRS5/Untitled.png

Still had lots of memory available.

I already used mprime as I posted before, it did not find anything.

Last edited by GloW_on_dub (2019-06-12 12:38:50)

seth · 2019-06-12 10:49:29

Please replace the oversized image w/ a thumbnail or link (the limit is 200x200)
Does it always bounce off the 6GB limit?

GloW_on_dub · 2019-06-12 14:26:08

Here is the memory profile for a build from scratch where the errors happend 5/6 times while still working and compiling other software on the same computer.
https://i.ibb.co/BZs19Mq/Untitled.png

Not sure what to make of it.

Last edited by GloW_on_dub (2019-06-12 14:26:42)

seth · 2019-06-12 14:33:12

Is the first instance at ~300 or at ~1300? The second one seems the most interesting section. The free memory went south and the system had to get it lots of the available memory so
a) there's indeed a massive demand.
b) maybe some of the avialable RAM wasn't all that available. So this could be a bug in the kernel or libc. => certainly try a different kernel

GloW_on_dub · 2019-06-12 16:34:34

This first issue was at 1300. But there were lots of them after this event where the memory change was not so drastic.
The massive demand is expected, I'm compiling a lot of files.

Your suggestion regarding kernel is interesting, but my colleagues using ArchLinux with somewhat similar computer do not encounter this.

seth · 2019-06-12 19:41:44

This would rather point back to the memory, maybe incompatibility among the modules.
What if you choose the very most conservative clockrates?

As for RAM tests, w/ 24GB meaningful memtest86 runs would take days, rather than hours.

GloW_on_dub · 2019-06-13 08:34:32

> What if you choose the very most conservative clockrates?
Not sure what you mean. I was able to do a full build without event using ninja -j 1.

> As for RAM tests, w/ 24GB meaningful memtest86 runs would take days, rather than hours.
Indeed, I will run mprime over the weekend.

rowdog · 2019-06-13 10:50:02

seth wrote:

As for RAM tests, w/ 24GB meaningful memtest86 runs would take days, rather than hours.

Ahem, uh, yeah, I feel old now. The last time I ran it I just left it on overnight and ... well, sorry for the dated advice.

rowdog · 2019-06-13 11:11:40

GloW_on_dub wrote:

> What if you choose the very most conservative clockrates?
Not sure what you mean. I was able to do a full build without event using ninja -j 1.
> As for RAM tests, w/ 24GB meaningful memtest86 runs would take days, rather than hours.
Indeed, I will run mprime over the weekend.

If you go into the BIOS and poke around a bit you should be able to find the CAS settings for your RAM. Make note of the current settings and try the one with the lowest numbers. A RAM stick that is marginal or mismatched may still work at lower speeds.

And sorry about the confusion, I believe that seth's remark about memtest86 was meant for me as I recommended that test.

seth · 2019-06-13 11:35:01

It was meant for everyone ;-)
You need to run several cycles (like a dozen?) and RAM amount grew much faster than RAM or CPU speeds

@GloW_on_dub
Also use memtest86 for RAM sanity checks rather than mprime.

GloW_on_dub · 2019-06-28 08:17:48

Here are the results for memtest86+:
http://scratchpad.io/annoyed-pump-5405

Interesting parts are :

Test 10 [Bit fade test, 2 patterns, 1 CPU]	0/4 (0%)	4
Test 13 [Hammer test]	0/4 (0%)	67

This is not so bad, but if I understand well, I'm not supposed to have any errors whatsoever.

I will start testing the RAM module independantly.

It seems to have identified a specific adress for the errors :

Highest/Lowest Error Address	0x252EED94C (9518MB)

This seems to correspond to a specific RAM modules considering they are listed in the right order here :

DIMM Slot #0	8GB 
DIMM Slot #1	4GB (this one ?)
DIMM Slot #2	8GB 
DIMM Slot #3	4GB

Is that correct ?

Last edited by GloW_on_dub (2019-06-28 08:20:49)

rowdog · 2019-06-28 10:21:02

Yes, any failure is bad. The results point to bad RAM in slot #1 but sometimes this type of result can come from mixing memory modules.
I'm just guessing but try swapping the RAM chips in slot #1 and #2. If the memory is interleaved, the banks need to be the same and usually slot #0 would be paired with slot #1. Same for 3 and 4. Or, maybe try using the 4gb chips in slots 0 and 1 and run memtest again to see if it still identifies the chip as faulty.

Edit: Really, you should check your motherboard book to see how RAM is laid out. I expect that RAM should be installed in banks rather than alternating.

Last edited by rowdog (2019-06-28 10:27:34)

GloW_on_dub · 2019-07-15 08:59:44

So I tested each RAM stick and one was bad indeed. Thanks for the help provided.

Arch Linux

#1 2019-06-06 14:25:44

[SOLVED]Random errors while compiling C++ code

#2 2019-06-06 14:44:00

Re: [SOLVED]Random errors while compiling C++ code

#3 2019-06-06 15:45:39

Re: [SOLVED]Random errors while compiling C++ code

#4 2019-06-06 15:53:17

Re: [SOLVED]Random errors while compiling C++ code

#5 2019-06-06 15:54:26

Re: [SOLVED]Random errors while compiling C++ code

#6 2019-06-06 16:04:42

Re: [SOLVED]Random errors while compiling C++ code

#7 2019-06-07 07:27:46

Re: [SOLVED]Random errors while compiling C++ code

#8 2019-06-07 13:09:01

Re: [SOLVED]Random errors while compiling C++ code

#9 2019-06-09 11:21:56

Re: [SOLVED]Random errors while compiling C++ code

#10 2019-06-09 21:01:57

Re: [SOLVED]Random errors while compiling C++ code

#11 2019-06-11 13:37:32

Re: [SOLVED]Random errors while compiling C++ code

#12 2019-06-12 10:00:20

Re: [SOLVED]Random errors while compiling C++ code

#13 2019-06-12 10:49:29

Re: [SOLVED]Random errors while compiling C++ code

#14 2019-06-12 14:26:08

Re: [SOLVED]Random errors while compiling C++ code

#15 2019-06-12 14:33:12

Re: [SOLVED]Random errors while compiling C++ code

#16 2019-06-12 16:34:34

Re: [SOLVED]Random errors while compiling C++ code

#17 2019-06-12 19:41:44

Re: [SOLVED]Random errors while compiling C++ code

#18 2019-06-13 08:34:32

Re: [SOLVED]Random errors while compiling C++ code

#19 2019-06-13 10:50:02

Re: [SOLVED]Random errors while compiling C++ code

#20 2019-06-13 11:11:40

Re: [SOLVED]Random errors while compiling C++ code

#21 2019-06-13 11:35:01

Re: [SOLVED]Random errors while compiling C++ code

#22 2019-06-28 08:17:48

Re: [SOLVED]Random errors while compiling C++ code

#23 2019-06-28 10:21:02

Re: [SOLVED]Random errors while compiling C++ code

#24 2019-07-15 08:59:44

Re: [SOLVED]Random errors while compiling C++ code

Board footer