[SOLVED] makepkg hangs sometimes recompiling kernel

archlinuxforever · 2018-09-21 14:46:49

Hi

I'm compiling kernel and sometimes makepkg hangs. This is the ps output after wait more tha one hour (recompiling use to last 15 minutes more or less):

[ ~]$ ps aux|grep pts/5
     2118  0.0  0.0  26048  5540 pts/5    S+   14:08   0:00 make -f ./scripts/Makefile.build obj=drivers/media need-builtin=1
     2565  0.0  0.0  25416  4684 pts/5    S+   14:08   0:00 make -f ./scripts/Makefile.build obj=drivers/media/pci need-builtin=1
     2596  0.0  0.0      0     0 pts/5    Z+   14:08   0:00 [make] <defunct>
     7412  0.0  0.0  26792  6036 pts/5    S+   14:10   0:00 make -f ./scripts/Makefile.build obj=drivers/parport need-builtin=
     7420  0.0  0.0      0     0 pts/5    Z+   14:10   0:00 [sh] <defunct>
     8310  0.0  0.0  10712  2188 pts/3    S+   15:26   0:00 grep pts/5
    24131  0.0  0.0  16476  6048 pts/5    S+   14:05   0:00 /usr/bin/bash /usr/bin/makepkg -fe
    24688  0.0  0.0  26180  5448 pts/5    S+   14:05   0:00 make bzImage modules
    25986  0.0  0.0  15416  5432 pts/5    Ss   12:44   0:00 bash
    26634  0.0  0.0  26064  5132 pts/5    S+   14:06   0:00 make -f ./scripts/Makefile.build obj=drivers need-builtin=1

This the output after see ps output and press CTRL+C:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe ; beep
==> Making package: linux-4.18-7 4.18.7.arch1-1 (Fri Sep 21 16:38:17 2018)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Removing existing $pkgdir/ directory...
==> Starting build()...
  DESCEND  objtool
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
^Cmake[1]: *** [scripts/Makefile.build:558: drivers/parport] Interrupt
make[2]: *** [scripts/Makefile.build:558: drivers/media/pci] Interrupt
make[1]: *** [scripts/Makefile.build:558: drivers/media] Interrupt
make: *** [Makefile:1033: drivers] Interrupt
==> ERROR: Aborted by user! Exiting...

I don't know why sometimes hangs. Any idea?

Regards

Last edited by archlinuxforever (2018-09-25 11:55:55)

archlinuxforever · 2018-09-23 19:26:17

Doing some research I think that is when I add

; beep

at end.

I have no idea why makepkg hangs when I add this at end.

jasonwryan · 2018-09-23 19:43:38

At the end of what?

Moving to NC...

archlinuxforever · 2018-09-23 20:37:26

At end of command.

If I write this command:

LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe ; beep

makepkg hangs, but if I write this command:

LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe

makepkg doesn't hangs.

Last edited by archlinuxforever (2018-09-23 20:38:03)

Trilby · 2018-09-23 20:44:06

Have you repeated that a few times to be sure it wasn't a coincidence, because there should be no way for the second command to have any effect at all on the former. If you are sure it's not a coincidence, does the same problem occur if you use a command other than beep, e.g., "echo done"?

seth · 2018-09-23 22:45:58

+1
Try beeping, but omit the MAKEFLAGS (ie. compile single jobbing). See whether that hangs as well.

archlinuxforever · 2018-09-24 11:31:42

I've research executing different commands in a row.

The first command in a row of four were run without problems:

[ testing-x86_64]$ LC_ALL=C makepkg -fe

The next command in a row of four were run without problems:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe

The next command in a row of four were run without problems:

[ testing-x86_64]$ LC_ALL=C makepkg -fe ; beep

The next command hangs at the secod time:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe ; beep

This is the output from ps:

[ ~]$ ps aux|grep pts/0
     1707  0.0  0.0  15420  5184 pts/0    Ss   07:53   0:00 bash
     2270  0.1  0.0  10952  3096 pts/1    S+   07:53   0:20 watch -dn5 ps aux|grep pts/0
     4967  0.2  0.0  28568  7612 pts/0    S+   12:02   0:00 make -f ./scripts/Makefile.build obj=drivers/net need-builtin=1
     5412  0.0  0.0  26496  5684 pts/0    S+   12:02   0:00 make -f ./scripts/Makefile.build obj=drivers/net/ethernet need-builtin=1
     6109  0.0  0.0      0     0 pts/0    Z+   12:03   0:00 [make] <defunct>
     6213  0.0  0.0  26344  5544 pts/0    S+   12:03   0:00 make -f ./scripts/Makefile.build obj=drivers/nfc need-builtin=1
     6234  0.0  0.0  26164  4928 pts/0    S+   12:03   0:00 make -f ./scripts/Makefile.build obj=drivers/nfc/microread need-builtin=
     6246  0.0  0.0      0     0 pts/0    Z+   12:03   0:00 [sh] <defunct>
     6447  0.0  0.0  10712  2200 pts/2    S+   12:04   0:00 grep pts/0
    24490  0.1  0.0  10608  5276 pts/0    S+   11:59   0:00 /usr/bin/bash /usr/bin/makepkg -fe
    25049  0.0  0.0  20564  4900 pts/0    S+   11:59   0:00 make bzImage modules
    27052  0.0  0.0  26064  5252 pts/0    S+   12:00   0:00 make -f ./scripts/Makefile.build obj=drivers need-builtin=1

The next command hangs at the third time:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe ; echo done

I don't know why adding "; beep" or "; echo done" to the command causes makepkg to hang sometimes when the MAKEFLAGS option is set.

seth · 2018-09-24 12:24:46

Try

$((`nproc`- 1))

Trilby · 2018-09-24 13:34:14

archlinuxforever wrote:

I don't know why adding "; beep" or "; echo done" to the command causes makepkg to hang

I don't know that adding those causes it to hang. From what you've presented here, it seems more parsimonious to suspect that makepkg might hang after a few runs regardless of the command line or perhaps related to repeated uses of makeflags but having nothing to do with the beep or echo commands.

Try rebooting and starting completely fresh and run the full command with make flags and ending in '; echo done'. Does this hang when it is used as the first invocation? If that doesn't hang, follow it up by repeating the command with makeflags but without any beep or echo at the end; do repeated uses of makepkg with the makeflags settings eventually hang (without beep or echo appended)?

In any case, when a hang eventually occurs, also check the output of `free -m` and `df -h` and post them here.

Last edited by Trilby (2018-09-24 13:36:50)

archlinuxforever · 2018-09-24 14:28:34

I repeated your example

$ LC_ALL=C MAKEFLAGS="-j$((`nproc`- 1))" makepkg -fe ; beep

in a row of four and runs without hang, but is not what I want.

I want to use my two cores (it will last half of time more or less) and a sound when it ends.

Last edited by archlinuxforever (2018-09-24 14:29:23)

seth · 2018-09-24 14:51:58

"Two"?

free -h

What happens for

LC_ALL=C MAKEFLAGS="-j4" makepkg -fe

(no beep, and that's perfectly fine, the job count isn't limited by your cores. This will hopefully just trigger faster whatever is happening there)

archlinuxforever · 2018-09-24 17:41:48

I ran four times the command without any problem:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j4" makepkg -fe ; beep

So the problem is use

MAKEFLAGS="-j$(nproc)"

and after makepkg add ; and a command like

; beep

or

; echo done

free -h returns:

[ ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7,8Gi       2,8Gi       593Mi       135Mi       4,5Gi       4,6Gi
Swap:         4,0Gi        79Mi       3,9Gi

seth · 2018-09-24 19:22:06

I don't think the ";beep" is any significant - it really cannot be.
My guess is that the limitation to exactly two jobs potentially leads to some sort of race condition between two jobs (waiting for a third one …) - and which does not occur on a linear build.

Are you using bash? What happens eg. on zsh?

archlinuxforever · 2018-09-25 11:55:10

Yes, I'm using bash. I tried with zsh and running:

% LC_ALL=C MAKEFLAGS="-j$(nproc)" makepkg -fe ; beep
==> Making package: linux-4.18-7 4.18.7.arch1-1 (Tue Sep 25 08:36:17 2018)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Removing existing $pkgdir/ directory...
==> Starting build()...
  DESCEND  objtool
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h

hangs at third time. I think that could be a race condition, but is weird that if I write -j4 it works. I tried eight times with zsh and it works without hang.

seth · 2018-09-25 12:37:48

The theoretical explanation would be some branched behavior.
Let's say there're jobs A and B which at some point rely on job C (and D, but C can be equal to D) being completed.
In the serial build, the system knows that it has to first complete job C before it can start jobs A & B.
In the parallel build, jobs A & B can be started and just have to stall for job C to complete.
The problem w/ -j2 is then, that the job queue is full and job C cannot be even started.

Of course that would amount to some sort of bug in the build system.

I can not come up w/ a theory on why the shell execution chain should have *any* impact on the behavior of the build system, thus I must keep rejecting that theory.
Especially since we seem to have evidence for the significance of the specific job count.

Trilby · 2018-09-25 14:12:28

How many cores do you *actually* have. I'm not sure if it would be considered a bug in the build system for the above condition to be met if you passed -jN where N was greater than your number of processor cores. If the build system was told that there should be 3 cores, for example, but there were really only two, it should be quite happy to start up A and B assuming it should still havea free core for C. But when that assumption proves wrong, it can never start C and A and B hang indefinitely.

Though this doesn't address why the build would work a couple times, then start failing.

seth · 2018-09-25 14:25:04

Erhhem… no.

The system is still multi-threaded.
You can run 100 parallel jobs on one logic core (it's not a good idea and you'll run OOM, but that's not relevant) - therefore the higher job count didn't cause the issue either.

The problem would arise if you artificially limited the amount of jobs and run into the situation where two parallel jobs start waiting for the completion of a thrid one to finish (and if those two jobs are not running at the same time, you'd end up w/ eg. jobs A&C, C completes, A can finish and B start on the slot formerly taken by C)

loqs · 2018-09-25 15:03:35

https://bugs.archlinux.org/task/58320 ?

archlinuxforever · 2018-09-26 09:39:37

Trilby wrote:

How many cores do you *actually* have. ...

I have 2 cores:

[ ~]$ LC_ALL=C lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          15
Model:               6
Model name:          Intel(R) Pentium(R) D CPU 3.40GHz
Stepping:            5
CPU MHz:             2400.000
CPU max MHz:         3400.0000
CPU min MHz:         2400.0000
BogoMIPS:            6823.88
L1d cache:           16K
L2 cache:            2048K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl cpuid pni dtes64 monitor ds_cpl est cid cx16 xtpr pdcm lahf_lm pti
[ ~]$

Researching I got this data:

With MAKEFLAGS="-j" tasks reach 1073 and it last about 12m/13m. I tested four times in a row without hang and it consumes about 1,5G more of RAM.

With MAKEFLAGS="-j4" last about 14m/15m.

With MAKEFLAGS="-j3" or MAKEFLAGS="-j$(($(nproc) + 1))" last about 13m, but hangs the third time:

[ testing-x86_64]$ LC_ALL=C MAKEFLAGS="-j$(($(nproc) + 1))" makepkg -fe ; beep

Whitout MAKEFLAGS it last about 20m.

The spanish version of archwiki https://wiki.archlinux.org/index.php/Ma … endaciones recommend to set -j to the number of cores plus one. This is not correct building the kernel.

The english version https://wiki.archlinux.org/index.php/Ma … ompilation says that could be race conditions if you change the MAKEFLAGS variable.

For me, the best option in my PC is only -j because it is faster, doesn't hangs and I can use my GNOME desktop meanwhile if I want.

Update: only -j is not a good option. I tried build kernel from scratch with only -j option and my PC with 8Gb gets out of memory. Compiling kernel hangs sometimes when you use the -j[num] option with different nums, I think that is better don't use it.

Last edited by archlinuxforever (2018-10-20 08:28:52)

Arch Linux

#1 2018-09-21 14:46:49

[SOLVED] makepkg hangs sometimes recompiling kernel

#2 2018-09-23 19:26:17

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#3 2018-09-23 19:43:38

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#4 2018-09-23 20:37:26

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#5 2018-09-23 20:44:06

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#6 2018-09-23 22:45:58

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#7 2018-09-24 11:31:42

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#8 2018-09-24 12:24:46

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#9 2018-09-24 13:34:14

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#10 2018-09-24 14:28:34

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#11 2018-09-24 14:51:58

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#12 2018-09-24 17:41:48

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#13 2018-09-24 19:22:06

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#14 2018-09-25 11:55:10

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#15 2018-09-25 12:37:48

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#16 2018-09-25 14:12:28

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#17 2018-09-25 14:25:04

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#18 2018-09-25 15:03:35

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

#19 2018-09-26 09:39:37

Re: [SOLVED] makepkg hangs sometimes recompiling kernel

Board footer