[SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

jamespharvey20 · 2018-10-19 11:12:25

Manually running the exact same commands in the PKGBUILD "prepare()" and "build()" compiles just fine. Running a PKGBUILD through "makepkg" causes a compilation failure in "build()". Or, through anything that uses "makepkg" like "devtools". But, here, I think "makepkg" better illustrates the problem, because it eliminates the chroot and ensures the manual vs PKGBUILD compilation are happening on identical packages.

OK, so it's the AUR's "gcc-git" PKGBUILD.

But, wait! You don't need to go to the next post. Yes, it's a huge and complex PKGBUILD. But, the failure happens in "build()", so most of the PKGBUILD is irrelevant. You can look at this reduced/gutted PKGBUILD that has the same problem, and is the size of a typical PKGBUILD. If you want to see the entire thing, here's the AUR PKGBUILD. If you want to see it, here's the core PKGBUiLD. The AUR's is a copy of core's, with as few changes as needed to use git trunk/master.

I'm the maintainer for AUR's "gcc-git", and right now it has to grab commit 7961f40b (2018-09-25) rather than the trunk/master. Letting it go to one commit beyond there to 81512c36 or anywhere after there causes a compilation failure in PKGBUILD "build()":

/build/gcc-git/src/gcc-build/./prev-gcc/xg++ -B/build/gcc-git/src/gcc-build/./prev-gcc/ -B/usr/x86_64-pc-linux-gnu/bin/ -nostdinc++ -B/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -B/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs  -I/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu  -I/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/include  -I/build/gcc-git/src/gcc/libstdc++-v3/libsupc++ -L/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/build/gcc-git/src/gcc-build/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -fno-PIE -c   -g -O2 -fchecking=1 -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H -I. -I. -I/build/gcc-git/src/gcc/gcc -I/build/gcc-git/src/gcc/gcc/. -I/build/gcc-git/src/gcc/gcc/../include -I/build/gcc-git/src/gcc/gcc/../libcpp/include  -I/build/gcc-git/src/gcc/gcc/../libdecnumber -I/build/gcc-git/src/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/build/gcc-git/src/gcc/gcc/../libbacktrace -I/build/gcc-git/src/gcc-build/./isl/include -I/build/gcc-git/src/gcc/isl/include -D_FORTIFY_SOURCE=2 -o insn-automata.o -MT insn-automata.o -MMD -MP -MF ./.deps/insn-automata.TPo insn-automata.c
/build/gcc-git/src/gcc/gcc/config/i386/i386.c: In function ‘const char* output_fix_trunc(rtx_insn*, rtx_def**, bool)’:
/build/gcc-git/src/gcc/gcc/config/i386/i386.c:19200:1: error: invalid operand in unary operation
19200 | output_fix_trunc (rtx_insn *insn, rtx *operands, bool fisttp)
      | ^~~~~~~~~~~~~~~~
/build/gcc-git/src/gcc/gcc/config/i386/i386.c:19200:1: error: incorrect sharing of tree nodes
(ssizetype) _19
_58 = (long unsigned int) ((ssizetype) _19 <= 7 ? 7 - (ssizetype) _19 : 0);
/build/gcc-git/src/gcc/gcc/config/i386/i386.c:19200: confused by earlier errors, bailing out
make[3]: *** [Makefile:2287: i386.o] Error 1

OK, so the offending commit means something changed upstream that requires the PKGBUILD to change, right?

I don't think so:
1. This failure happens in gcc's bootstrap stage 3. If it compiled in stage 2, it should compile in stage 3. It's just a do-over. (Stage 1 uses system gcc to compile new gcc, stage 2 uses new gcc to compile itself to get new optimizations, and stage 3 compiles itself again and compares for verification.) Often if someone references a stage 3 failure, they're talking about the comparison between stages 2 and 3 - which is not what's failing here.
2. Again, I can use the offending commit (or anything more recent, including the most recent commit bf39a88f) and copy/paste the commands from the PKGBUILD into bash, and it compiles fine.
3. If you look at the offending commit, it doesn't strike me as something that would affect configure options, dependencies, or anything in the PKGBUILD.

OK, so the offending commit means upstream is broken?

I don't think so:
1. Again, the compilation errors only when "makepkg" is involved.
2. The commit was made 3 weeks ago, and I can't find anyone else mentioning this failure.
3. Again, only fails in stage 3.

Again, the AUR PKGBUILD is as close as possible to core's PKGBUILD. "prepare()" and "build()" are identical, other than the AUR's leaves off "--enable-libmpx" from configure, because gcc removed libmpx 4 months ago. (That's 3 months before the offending commit, so I don't see that having anything to do with this problem.)

So, here's what core's and AUR's PKGBUILD do, in "prepare()" and "build()" :

  [[ ! -d gcc ]] && ln -s gcc-${pkgver/+/-} gcc
  cd gcc

  # link isl for in-tree build
  ln -s ../isl-${_islver} isl

  # Do not run fixincludes
  sed -i 's@\./fixinc\.sh@-c true@' gcc/Makefile.in

  # Arch Linux installs x86_64 libraries /lib
  sed -i '/m64=/s/lib64/lib/' gcc/config/i386/t-linux64

  # hack! - some configure tests for header files using "$CPP $CPPFLAGS"
  sed -i "/ac_cpp=/s/\$CPPFLAGS/\$CPPFLAGS -O2/" {libiberty,gcc}/configure

  mkdir -p "$srcdir/gcc-build"

  cd gcc-build

  # using -pipe causes spurious test-suite failures
  # http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48565
  CFLAGS=${CFLAGS/-pipe/}
  CXXFLAGS=${CXXFLAGS/-pipe/}

  "$srcdir/gcc/configure" --prefix=/usr \
      --libdir=/usr/lib \
      --libexecdir=/usr/lib \
      --mandir=/usr/share/man \
      --infodir=/usr/share/info \
      --with-bugurl=https://bugs.archlinux.org/ \
      --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++ \
      --enable-shared \
      --enable-threads=posix \
      --with-system-zlib \
      --with-isl \
      --enable-__cxa_atexit \
      --disable-libunwind-exceptions \
      --enable-clocale=gnu \
      --disable-libstdcxx-pch \
      --disable-libssp \
      --enable-gnu-unique-object \
      --enable-linker-build-id \
      --enable-lto \
      --enable-plugin \
      --enable-install-libiberty \
      --with-linker-hash-style=gnu \
      --enable-gnu-indirect-function \
      --enable-multilib \
      --disable-werror \
      --enable-checking=release \
      --enable-default-pie \
      --enable-default-ssp \
      --enable-cet=auto

  make

Outside of a PKGBUILD, we can download the upstream git trunk source, and manually perform these exact same commands. It builds just fine.

$ git clone https://gcc.gnu.org/git/gcc.git
$ wget http://isl.gforge.inria.fr/isl-0.19.tar.bz2
$ tar xvjf isl-0.19.tar.bz2
$ (perform all the steps above, from PKGBUILD, of course using "ln -s ../isl-0.19 isl" instead of referencing a non-existent variable)

I can't help but notice this fails in "gcc/config/i386/i386.c", and core's and AUR's PKGBUILD performs "sed -i '/m64=/s/lib64/lib/' gcc/config/i386/t-linux64". I don't think this is related, though:
1. The offending commit doesn't touch this file. Notably, 7 commits and 13 commits ahead of the offending commit, "config/i386/i386.md" is modified. Also notably, 5 commits after the offending commit is named "runtime, os: fix the build on Solaris", but its changes appear completely unrelated and don't mention this problem.
2. But, again, it compiles outside of "makepkg".
2. And, again, it compiles within "makepkg" in stage 2, and only fails in the stage 3 redo.

No behavior changes depending on if parallel/non-parallel make is used. Meaning, "make" vs "make -j" with manual commands, and whether 'MAKEFLAGS="-j#"' is in "/etc/makepkg.conf" for "makepkg" or anything using it.

If this is a hardware issue, it's the most obscure one I've ever seen. Kernel compiles fine. System is rock solid. Memtest passes 16+ hours. I found the offending commit through a git bisect. Bisecting between "gcc-8_2_0-release" and current "trunk/master" has a bunch of steps. Everything after the offending commit fails, everything before succeeds. There were no "that doesn't make sense" moments. (That said, I still will like to hear someone say they can reproduce what I'm seeing.)

Is "makekpg" somehow at fault here?

If people say no, I'll be glad to go upstream. But, I assume they are going to say it's an Arch issue, given that everyone else has been compiling it fine for 3 weeks, and it works manually on Arch, just not through "makepkg"? I'd love an argument to counter that.

EDIT: Wanted to add that I'm not interested in adding the "--disable-bootstrap" configure option. I don't want to run a gcc that fails bootstrap stage 3, and don't feel comfortable releasing a PKGBUILD using the option.

Last edited by jamespharvey20 (2018-10-20 02:09:03)

hpmachining · 2018-10-19 12:32:18

If it builds manually but not with makepkg it is probably one of the CFLAGS, CXXFLAGS, etc. in makepkg.conf that is causing the build to fail.

jasonwryan · 2018-10-20 01:55:48

Reopened...

jamespharvey20 · 2018-10-20 02:08:33

hpmachining wrote:

If it builds manually but not with makepkg it is probably one of the CFLAGS, CXXFLAGS, etc. in makepkg.conf that is causing the build to fail.

You're right, thank you.

I tested this and determined it wasn't the cause, but apparently I tested it in the wrong direction. I took the CPPFLAGS/CFLAGS/CXXFLAGS/LDFLAGS set in "/etc/makepkg.conf" and set those, before manually executing the commands in the PKGBUILD, and it failed. That seemed like a good test to me.

Just now, testing the other way, adding "!buildflags" to PKGBUILD "options=", "makepkg" succeeds with trunk/master.

So, I'm still baffled why gcc bootstrap stage 2 succeeds with the "/etc/makepkg.conf' flags but when it repeats it fails in stage 3... But, that very strongly points to an upstream issue.

EDIT: Sigh. Examining my bash history, the flags weren't exported, so configure never got them, so when it succeeded, it did so without the flags. All is sane now. Exporting the default makepkg flags makes it fail outside of makepkg. Definite upstream issue.

Last edited by jamespharvey20 (2018-10-20 04:02:41)

eschwartz · 2018-10-21 02:56:47

Have you tried to figure out the minimal subset of those flags, which causes the issue to manifest?

jamespharvey20 · 2018-10-21 03:29:35

eschwartz wrote:

Have you tried to figure out the minimal subset of those flags, which causes the issue to manifest?

Just finished, actually. Outside of "makepkg", this reproduces a build failure, and seems to be as reduced as I can get it:

$ cd <gcc git clone dir>
$ sed -i "/ac_cpp=/s/\$CPPFLAGS/\$CPPFLAGS -O2/" {libiberty,gcc}/configure
$ mkdir ../build
$ cd ../build
$ export CPPFLAGS="-D_FORTIFY_SOURCE=2"
$ ../<gcc git clone dir>/configure --disable-werror

I should have an upstream gcc bug posted shortly, and will link here once up.

"-D_FORTIFY_SOURCE=2" (from /etc/makepkg.conf) of course requires "-O2" to run. Core's gcc PKGBUILD has added the "-O2" flags to those two configure files since 4.8.0-2 (2013-04-12.)

I don't know if any other distributions do this. This probably explains why others building gcc from git aren't having the problem. If you even have one without the other, I don't think the build fails since "D_FORTIFY_SOURCE=2" isn't done in that case.

In my bugreport, I was hoping to be able to change "{libiberty,gcc}/configure" to just "gcc/configure". But, doing so gives an almost immediate build failure: (much nicer to deal with than in stage 3...)

../../gcc.mirror.git.local/libiberty/fibheap.c:38:24: error: ‘LONG_MIN’ undeclared (first use in this function)
 #define FIBHEAPKEY_MIN LONG_MIN
                        ^~~~~~~~

I'm going to ignore that for now and just leave it changing both files.

"--disable-werror" is needed at the moment, or the build fails elsewhere on warnings. I think it's on fscanf return values being ignored.

We'll see if their response is simply to stop modifying these configure files and using "D_FORTIFY_SOURCE=2". I hope not. I see this as Arch tests/stresses gcc's build a bit more than typically done, and by doing so, it uncovered something that's wrong. EDIT: Besides that, whatever you throw at gcc shouldn't cause an internal compiler error, which started showing up as a third error spit out at the same time, in addition to the 2 I posted before once I pruned flags and/or configure options away.

It's definitely at that commit, and current master still reproduces.

Last edited by jamespharvey20 (2018-10-21 03:42:49)

jamespharvey20 · 2018-10-21 04:22:47

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87672

EDIT: This bugreport has a patch included in it that appears to fix the problem. It has not yet been committed, but it's added in AUR gcc-git so it's able to use top trunk/master again.

Last edited by jamespharvey20 (2018-10-24 08:28:01)

Arch Linux

#1 2018-10-19 11:12:25

[SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#2 2018-10-19 12:32:18

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#3 2018-10-20 01:55:48

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#4 2018-10-20 02:08:33

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#5 2018-10-21 02:56:47

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#6 2018-10-21 03:29:35

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

#7 2018-10-21 04:22:47

Re: [SOLVED] Manual compile works, PKGBUILD fails with makepkg / devtools

Board footer