What is happening with glibc and gcc packages?

johnyan · 2022-02-07 21:29:09

Hey @Toolybird, thanks for sharing the link. I think I am almost down to the bottom of it.

sha1_finish_ctx is defined in sha1.c.

I think the issue is the -flto gcc flag.

[root@john libiberty]# gcc -c -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -flto      -I. -I/build/binutils/src/binutils-gdb/libiberty/../include  -W -Wall -Wwrite-strings -Wc++-compat -Wstrict-prototypes -Wshadow=local -pedantic  -D_GNU_SOURCE -fcf-protection /build/binutils/src/binutils-gdb/libiberty/sha1.c -o sha1.o
[root@john libiberty]# objdump -D sha1.o | grep sha1_finish_ctx
Disassembly of section .gnu.lto_sha1_finish_ctx.52.506c201690acd78c:
0000000000000000 <.gnu.lto_sha1_finish_ctx.52.506c201690acd78c>:
[root@john libiberty]# gcc -c -DHAVE_CONFIG_H -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection      -I. -I/build/binutils/src/binutils-gdb/libiberty/../include  -W -Wall -Wwrite-strings -Wc++-compat -Wstrict-prototypes -Wshadow=local -pedantic  -D_GNU_SOURCE -fcf-protection /build/binutils/src/binutils-gdb/libiberty/sha1.c -o sha1.o
[root@john libiberty]# objdump -D sha1.o | grep sha1_finish_ctx
00000000000013f0 <sha1_finish_ctx>:
   ...

As you can see, with -flto, the sha1_finish_ctx will get transformed to .gnu.lto_sha1_finish_ctx.52.506c201690acd78c and also becomes a non-function.
without -flto, the name is correct.

Adding these before ./configure should work.

  # remove lto
  CFLAGS=${CFLAGS/-flto/}
  CXXFLAGS=${CXXFLAGS/-flto/}
  LDFLAGS=${LDFLAGS/-flto/}

EDIT: yes, it works wonderfully.

Last edited by johnyan (2022-02-07 22:40:37)

Allan · 2022-02-07 23:44:46

johnyan wrote:

  # remove lto
  CFLAGS=${CFLAGS/-flto/}
  CXXFLAGS=${CXXFLAGS/-flto/}
  LDFLAGS=${LDFLAGS/-flto/}

Use this instead:

options=('!lto')

My entire toolchain is built with that specified in the PKGBUILD. I'm looking at adding --enable-pgo-build=lto to both binutils and gcc once the new binutils release is out and I have merged my gcc-12 pre-release branch.

johnyan · 2022-02-07 23:52:40

Allan wrote:

My entire toolchain is built with that specified in the PKGBUILD. I'm looking at adding --enable-pgo-build=lto to both binutils and gcc once the new binutils release is out and I have merged my gcc-12 pre-release branch.

Ahh, thanks for the tip. PKGCONFIG newbies here

johnyan · 2022-02-08 02:58:26

I got a bunch of test failures on gcc. Now I wonder how are you guys deal with gcc in the past? especially how do you maintain patches between major releases? Since 11.2.0 was released, there're patches added to upstream almost every day. Are you going to keep adding all those patches on top?

@Allan @Xyne - you guys are using completely different patches for gcc 11.2. May I ask where did you get those patches from?

Thanks.

Allan · 2022-02-08 03:20:23

johnyan wrote:

@Allan @Xyne - you guys are using completely different patches for gcc 11.2. May I ask where did you get those patches from?

The source of my patches are documented in the PKGBUILD. The SSP/PIE failure ones were written by me - SSP has been accepted upstream. The analyzer failures patch was submitted upstream by someone else.

Looks like Xyne pulled the patches from the current Arch GCC PKGBUILD. Some were irrelevant for my reduced toolchain (e.g. Ada), others I don't need as I am building from newer sources.

johnyan wrote:

Since 11.2.0 was released, there're patches added to upstream almost every day. Are you going to keep adding all those patches on top?

I do a full toolchain rebuild at least every month, so the patches on the relevant branch get included.

johnyan · 2022-02-08 03:28:44

I guess the question is that what's the policy to handle upstream patches?

1. Accept all patches - which might require frequent rebuilds.
2. Ignore all of them until next release.
3. Accept some of the patches - How do you determine if one is needed or not? policies?

Thanks.

Allan · 2022-02-08 03:48:15

johnyan wrote:

I guess the question is that what's the policy to handle upstream patches?
1. Accept all patches - which might require frequent rebuilds.
2. Ignore all of them until next release.
3. Accept some of the patches - How do you determine if one is needed or not? policies?

Arch does not have a policy. Some like building from the head of the git release branch as you get all patches and avoid the pain of backports (mostly). Others like to backport patches that fix specific problems.

I think using the head of the git branch is best, because it does include many fixes that upstream developers agree with, and so should not miss fixes for more minor issues that you are not fully aware of to backport. However... I once did that for binutils, and the linux kernel compiled with that patch corrupted all ext3/4 filesystems on boot beyond recovery. So there are some risks!

As to the frequency of updates if building from the branch, a practical approach is needed - rebuild when it is needed and the packager has time to fix any issues.

johnyan · 2022-02-08 15:24:33

Yeah, it makes sense. Found around 200 failures on gcc check on 11.2.0 release tar ball. I might try the tip of the gcc 11.2 release branch instead.

Toolybird · 2022-02-08 21:01:22

Xyne wrote:

I've uploaded the toolchain patches with those patches applied here along with a modified version of Allan's bootstrap script.

Hey @Xyne, thanks for doing all of that. Nifty scripts! But I was wondering about this item on the TODO:

> Merge gcc-libs back into gcc to avoid having to split the package in the PKGBUILD (cmp. Allan's toolchain)

@Allan, I've read your rationale, but I'm wondering is this something we really want? For sure the current PKGBUILD is ugly AF, but are we doing this solely to simplify the PKGBUILD? At one point I actually re-wrote the GCC PKGBUILD to make all the splits less yuck. Yeah, pkg splits are not really the Arch way, but they do have their uses. I guess I have a slight philosophical issue with pulling in the whole compiler as a runtime dep. Anyway, something to think about and maybe discuss.

johnyan · 2022-02-08 21:31:10

Currently still seeing a few failures:

               === g++ tests ===


Running target unix
FAIL: g++.dg/gcov/gcov-17.C test-gcov-17.py::test_basics - FileNotFoundError: [Errno 2] No such file or directory: 'gcov-17.C.gcov.json.gz
FAIL: g++.dg/gcov/gcov-17.C test-gcov-17.py::test_lines - FileNotFoundError: [Errno 2] No such file or directory: 'gcov-17.C.gcov.json.gz
FAIL: g++.dg/gcov/pr98273.C test-pr98273.py::test_basics - FileNotFoundError: [Errno 2] No such file or directory: 'pr98273.C.gcov.json.gz
FAIL: g++.dg/gcov/pr98273.C test-pr98273.py::test_lines - FileNotFoundError: [Errno 2] No such file or directory: 'pr98273.C.gcov.json.gz

                === g++ Summary ===

# of expected passes            208891
# of unexpected failures        4
# of expected failures          1101
# of unsupported tests          9168

               === gotools tests ===

FAIL: go test cmd/go
FAIL: TestScript
FAIL: go test runtime
FAIL: TestTinyAllocIssue37262

                === gotools Summary ===
# of expected passes            394
# of unexpected failures        4
# of untested testcases         124

Running target unix/-m32
FAIL: net
FAIL: runtime
FAIL: time

                === libgo Summary for unix/-m32 ===

# of expected passes            181
# of unexpected failures        3

                === libstdc++ tests ===


Running target unix
FAIL: 20_util/from_chars/4.cc execution test
FAIL: 20_util/from_chars/5.cc execution test

                === libstdc++ Summary ===

# of expected passes            14271
# of unexpected failures        2
# of expected failures          104
# of unsupported tests          659

Allan · 2022-02-08 21:39:02

Toolybird wrote:

@Allan, I've read your rationale, but I'm wondering is this something we really want?

"We" being the distro - no.
"We" being people that have the compiler installed - yes

Xyne · 2022-02-08 22:50:59

Allan wrote:

I once did that for binutils, and the linux kernel compiled with that patch corrupted all ext3/4 filesystems on boot beyond recovery. So there are some risks!

Well f***, that's scary.

Toolybird wrote:

> Merge gcc-libs back into gcc to avoid having to split the package in the PKGBUILD (cmp. Allan's toolchain)
@Allan, I've read your rationale, but I'm wondering is this something we really want? For sure the current PKGBUILD is ugly AF, but are we doing this solely to simplify the PKGBUILD? At one point I actually re-wrote the GCC PKGBUILD to make all the splits less yuck. Yeah, pkg splits are not really the Arch way, but they do have their uses. I guess I have a slight philosophical issue with pulling in the whole compiler as a runtime dep. Anyway, something to think about and maybe discuss.

I don't have a strong opinion either way. I was a little tired when I wrote it and my eyes were glossing over after reading through the PKGBUILD so I was more inclined towards ripping it all out and starting with something simpler. I'd be interested in seeing your less sucky splits.

I haven't touched this since I posted the repo. I got side-swiped by real-life obligations. I hope to have some time towards the end of the week.

Allan · 2022-02-08 23:11:32

Just gcc-libs vs gcc could be a compromise. Though there is a bug report asking for gcc-go to be split further, so this may not go down well...

johnyan · 2022-02-08 23:41:39

Some of the go test require "127.0.0.1 localhost" being add to /etc/hosts. I assume they should be ignore?

johnyan · 2022-02-09 13:32:23

by using GCC default build flags "-g -O2" as documented here: https://gcc.gnu.org/install/build.html, Here's my latest result:

                === g++ tests ===                                                               
                                                                                                

Running target unix
FAIL: g++.dg/gcov/gcov-17.C test-gcov-17.py::test_basics - FileNotFoundError: [Errno 2] No such file or directory: 'gcov-17.C.gcov.json.gz
FAIL: g++.dg/gcov/gcov-17.C test-gcov-17.py::test_lines - FileNotFoundError: [Errno 2] No such file or directory: 'gcov-17.C.gcov.json.gz
FAIL: g++.dg/gcov/pr98273.C test-pr98273.py::test_basics - FileNotFoundError: [Errno 2] No such file or directory: 'pr98273.C.gcov.json.gz
FAIL: g++.dg/gcov/pr98273.C test-pr98273.py::test_lines - FileNotFoundError: [Errno 2] No such file or directory: 'pr98273.C.gcov.json.gz

                === libgo Summary for unix ===

# of expected passes            184

Running target unix/-m32
FAIL: net
FAIL: runtime
FAIL: time

                === libgo Summary for unix/-m32 ===

# of expected passes            181
# of unexpected failures        3

For the 'FAIL: net' test, there's no real failures as the chroot env doesn't have localhost setup in /etc/hosts

For the 'Fail: time' test, here's the error:

--- FAIL: TestDurationSeconds (0.00s)
    time_test.go:1095: Duration(300ms).Seconds() = 0; want: 0.3
--- FAIL: ExampleParseDuration (0.00s)
got:
10h0m0s
1h10m10s
There are 4210 seconds in 1h10m10s.
There are 1000 nanoseconds in 1µs.
There are 0.00e+00 seconds in 1µs.
want:
10h0m0s
1h10m10s
There are 4210 seconds in 1h10m10s.
There are 1000 nanoseconds in 1µs.
There are 1.00e-06 seconds in 1µs.
FAIL
FAIL: time

for the 'FAIL: runtime', here's the error:

--- FAIL: TestGoroutineParallelism2 (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x567b406b]

Any suggestions/ideas how to fix those?

Here's my latest PKGBUILD: https://github.com/john-yan/arch-linux- … c/PKGBUILD

Thanks.

Allan · 2022-02-09 14:33:12

My guess is that is your build environment setup and you can ignore it.

johnyan · 2022-02-09 17:34:58

Thanks Allan.

I continue to do the following:

1. asp checkout and rebuild the linux repo, it seems successful. But there's no test.
2. rebuild the libtool repo. there is 2 test failures:

## ------------------------ ##
## Summary of the failures. ##
## ------------------------ ##
Failed tests:
GNU Libtool 2.4.6.42-b88ce-dirty test suite test groups:

 NUM: FILE-NAME:LINE     TEST-GROUP-NAME
      KEYWORDS

  67: link-order2.at:46  Link order of deplibs
      libtool interactive
 170: cmdline_wrap.at:28 Run tests with low max_cmd_len
      recursive expensive

3. rebuild valgrind and it also seems successful

Is the two test failure on libtool concern anyone?

johnyan · 2022-02-09 18:07:00

I just pushed all my change into my repo here: https://github.com/john-yan/arch-linux-toolchain

Allan, would you please let me know how to submit to arch for a code review?

Thanks.

agapito · 2022-02-09 20:57:31

¡Good news!

GCC: https://archlinux.org/packages/testing/x86_64/gcc/

GLIBC: https://archlinux.org/packages/testing/x86_64/glibc/

BINUTILS: https://archlinux.org/packages/testing/x86_64/binutils/

Last edited by agapito (2022-02-09 23:16:26)

Allan · 2022-02-09 21:10:49

johnyan wrote:

Allan, would you please let me know how to submit to arch for a code review?

There has never been any way to do that. An the dev team had working packages last week/month but were waiting for binutils to release.

BTW, your libtool failures are because that test can not use LTO. Fedora has a patch which has been pulled to Arch.

loqs · 2022-02-09 21:39:42

agapito wrote:

GLIBC: https://archlinux.org/packages/staging/x86_64/glibc/

Sadly with CET still enabled [1].

[1] https://lore.kernel.org/all/a7e59ae16e0 … intel.com/

johnyan · 2022-02-09 21:41:27

Ahh, glad to see things are moving again!

agapito · 2022-02-09 22:08:16

I've installed the new toolchain and my computer hasn't exploded yet, I am not having any problems at the moment.

I could compile my custom kernel just fine, but other packages failed to compile throwing these kind of errors

/usr/bin/ld: /usr/lib/libx265.so.199: reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21' not defined

Last edited by agapito (2022-02-09 22:11:44)

Allan · 2022-02-09 23:12:44

loqs wrote:

agapito wrote:
GLIBC: https://archlinux.org/packages/staging/x86_64/glibc/
Sadly with CET still enabled [1].
[1] https://lore.kernel.org/all/a7e59ae16e0 … intel.com/

Any chance you want to open a bug report with more explanation than a very long mailing list link?

johnyan · 2022-02-09 23:17:03

Also the new toolchain doesn't move the --enable-install-libiberty to binutils. Do you think it's worth to open a bug report? I installed the new staging toolchain. so far no issue.

Arch Linux

#126 2022-02-07 21:29:09

Re: What is happening with glibc and gcc packages?

#127 2022-02-07 23:44:46

Re: What is happening with glibc and gcc packages?

#128 2022-02-07 23:52:40

Re: What is happening with glibc and gcc packages?

#129 2022-02-08 02:58:26

Re: What is happening with glibc and gcc packages?

#130 2022-02-08 03:20:23

Re: What is happening with glibc and gcc packages?

#131 2022-02-08 03:28:44

Re: What is happening with glibc and gcc packages?

#132 2022-02-08 03:48:15

Re: What is happening with glibc and gcc packages?

#133 2022-02-08 15:24:33

Re: What is happening with glibc and gcc packages?

#134 2022-02-08 21:01:22

Re: What is happening with glibc and gcc packages?

#135 2022-02-08 21:31:10

Re: What is happening with glibc and gcc packages?

#136 2022-02-08 21:39:02

Re: What is happening with glibc and gcc packages?

#137 2022-02-08 22:50:59

Re: What is happening with glibc and gcc packages?

#138 2022-02-08 23:11:32

Re: What is happening with glibc and gcc packages?

#139 2022-02-08 23:41:39

Re: What is happening with glibc and gcc packages?

#140 2022-02-09 13:32:23

Re: What is happening with glibc and gcc packages?

#141 2022-02-09 14:33:12

Re: What is happening with glibc and gcc packages?

#142 2022-02-09 17:34:58

Re: What is happening with glibc and gcc packages?

#143 2022-02-09 18:07:00

Re: What is happening with glibc and gcc packages?

#144 2022-02-09 20:57:31

Re: What is happening with glibc and gcc packages?

#145 2022-02-09 21:10:49

Re: What is happening with glibc and gcc packages?

#146 2022-02-09 21:39:42

Re: What is happening with glibc and gcc packages?

#147 2022-02-09 21:41:27

Re: What is happening with glibc and gcc packages?

#148 2022-02-09 22:08:16

Re: What is happening with glibc and gcc packages?

#149 2022-02-09 23:12:44

Re: What is happening with glibc and gcc packages?

#150 2022-02-09 23:17:03

Re: What is happening with glibc and gcc packages?

Board footer