Test for AWS-LC fails - How to debug

lquidfire · 2025-06-25 13:08:57

Hi!

I am trying to build AWS-LC.

I am using the following incantations, based on their documentation, as well as the Archlinux wiki for Cmake PKGBUILDs:

$ cmake -GNinja -B build \
            -DCMAKE_BUILD_TYPE=RelWithDebInfo \
            -DCMAKE_INSTALL_PREFIX=/usr \
            -DCMAKE_INSTALL_SBINDIR:PATH=bin \
            -DBUILD_SHARED_LIBS=1

$ ninja -C build

So far, so good.

$ ninja -C build run_tests

Towards the end of the CI tests to be run, it hangs. The last line is always:

ssl/integration_test

I can't seem to find out where exactly, or why it hangs. I ran the run_tests through gdb, and I couldn't find the issue (I don't usually wrangle with gdb output).

The output of gdb (with interrupt and backtrace) can be found here: http://0x0.st/8ljC.txt

When I manually go into the directory "build/ssl" and then run "./integration_test", it runs just fine (and passes all tests).

Update: In fact, all the tests run fine if invoked manually.

Perhaps the run_tests programme crashes after this test, or during another test, but I don't know how to find out. Adding "-v" to the ninja command did not add more information.

I hope someone can help.

Thank you in advance!

________

Looking through the gdb output, I notice an error:

        parser = {<Parser> = {_vptr.Parser = 0x55555559e688 <vtable for ManifestParser+16>, state_ = 0x7fffffffdc68, file_reader_ = 0x7fffffffdd70, lexer_ = {filename_ = {str_ = 0x7fffffffdb20 "build.ni", len_ = 11}, input_ = {str_ = 0x7ffff783d010 <error: Cannot access memory at address 0x7ffff783d010>, len_ = 893097}, ofs_ = 0x7ffff79170ba <error: Cannot access memory at address 0x7ffff79170ba>, last_token_ = 0x7ffff79170b9 <error: Cannot access memory at address 0x7ffff79170b9>}}, env_ = 0x7fffffffdce8, options_ = {phony_cycle_action_ = kPhonyCycleActionWarn}, quiet_ = false}

However, grepping through the directory for "build.ni" (in case that is a wrong filename and should be "build.ini") does not give any results. Perhaps I'm on the wrong track.

There are a number of occurrences of "optimized out", so, for what it's worth, I will try build this with -O0 when I am home again.

Last edited by lquidfire (2025-06-26 09:30:30)

Lone_Wolf · 2025-06-26 09:38:07

This topic would fit better in Creating & Modifying Packages OR Programming & Scripting .

Do you intend to create a PKGBUILD for this ?

lquidfire · 2025-06-26 09:39:01

If I can get it built and working, then yes. There are packages for LibreSSL and WolfSSL.

Lone_Wolf · 2025-06-26 09:59:22

The approach that tends to work best on archlinux is to start a PKGBUILD.
One of the reasons for that is upstream usually targets debian/ubuntu and fedora/redhat type distros and archlinux does things different .

Moderator Note
Moving to Creating & Modifying Packages

lquidfire · 2025-06-26 10:14:06

Thanks.

A first attempt at a PKGBUILD is here: http://0x0.st/8l47.txt

Since the errors I tried to build and test the package manually, hoping there would be more error output.

Last edited by lquidfire (2025-06-26 10:14:41)

Lone_Wolf · 2025-06-26 19:28:48

When building in a clean chroot crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=BNTest.DISABLED_WycheproofPrimality hung .
I gave it 10 minutes, then aborted build .

log of the check() function

ninja: Entering directory `build'
[0/2] cd /build/aws-lc/src/aws-lc-1.53.1 && /usr/bin/cmake -E echo Running\ SSL\ tests && cd ssl/test/runner && /usr/bin/go test -timeout 10m -shim-path /build/aws-lc/src/aws-lc-1.53.1/build/ssl/test/bssl_shim -handshaker-path /build/aws-lc/src/aws-lc-1.53.1/build/ssl/test/handshaker -ssl-transfer-test-file /build/aws-lc/src/aws-lc-1.53.1/ssl/test/runner/ssl_transfer/test_case_names.txt
Running SSL tests
go: downloading golang.org/x/crypto v0.10.0
go: downloading golang.org/x/sys v0.9.0
0/0/976/1000/9321
                  0/0/977/1000/9321
                  0/0/1976/2000/9321
                   0/0/1976/2000/9321
                   0/0/1977/2000/9321
                   0/0/2976/3000/9321
                   0/0/2977/3000/9321
                   0/0/3976/4000/9321
                   0/0/3976/4000/9321
                   0/0/3976/4000/9321
                   0/0/3977/4000/9321
                   0/0/4976/5000/9321
                   0/0/4977/5000/9321
                   0/0/5976/6000/9321
                   0/0/5977/6000/9321
                   0/0/6976/7000/9321
                   0/0/6977/7000/9321
                   0/0/7976/8000/9321
                   0/0/7976/8000/9321
                   0/0/7977/8000/9321
                   0/0/8976/9000/9321
                   0/0/8977/9000/9321
                   0/0/9321/9321/9321

PASS
ok  	boringssl.googlesource.com/boringssl/ssl/test/runner	7.845s
[1/2] cd /build/aws-lc/src/aws-lc-1.53.1 && /usr/bin/cmake -E echo && /usr/bin/cmake -E echo Running\ Go\ tests && /usr/bin/go test ./ssl/test/runner/hpke ./util/ar && /usr/bin/cmake -E echo && /usr/bin/cmake -E echo Running\ unit\ tests && /usr/bin/go run util/all_tests.go -build-dir /build/aws-lc/src/aws-lc-1.53.1/build

Running Go tests
ok  	boringssl.googlesource.com/boringssl/ssl/test/runner/hpke	0.098s
ok  	boringssl.googlesource.com/boringssl/util/ar	0.004s

Running unit tests
crypto/crypto_test [shard 2/24]
crypto/crypto_test [shard 7/24]
crypto/crypto_test [shard 5/24]
crypto/crypto_test --gtest_filter=Crypto.OnDemandIntegrityTest (custom environment)
crypto/urandom_test
crypto/urandom_test (custom environment)
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=SnapsafeGenerationTest.*
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/crypto_test --fork_unsafe_buffering --gtest_filter=RandTest.*:-RandTest.Fork
crypto/crypto_test --gtest_filter=RandTest.* (custom environment)
crypto/crypto_test [shard 6/24]
crypto/crypto_test --gtest_filter=ForkDetect.* (custom environment)
crypto/crypto_test [shard 19/24]
crypto/crypto_test [shard 9/24]
crypto/crypto_test [shard 15/24]
crypto/crypto_test [shard 17/24]
ssl/ssl_test [shard 1/24]
ssl/ssl_test [shard 5/24]
crypto/crypto_test [shard 11/24]
ssl/ssl_test [shard 2/24]
crypto/crypto_test [shard 13/24]
ssl/ssl_test [shard 3/24]
ssl/ssl_test [shard 6/24]
crypto/crypto_test [shard 16/24]
ssl/ssl_test [shard 4/24]
ssl/ssl_test [shard 7/24]
ssl/ssl_test [shard 10/24]
ssl/ssl_test [shard 9/24]
ssl/ssl_test [shard 8/24]
ssl/ssl_test [shard 11/24]
ssl/ssl_test [shard 12/24]
ssl/ssl_test [shard 13/24]
crypto/crypto_test [shard 3/24]
ssl/ssl_test [shard 14/24]
ssl/ssl_test [shard 16/24]
ssl/ssl_test [shard 17/24]
crypto/mem_test
crypto/mem_set_test
crypto/dynamic_loading_test
ssl/ssl_test [shard 15/24]
crypto/rwlock_static_init
ssl/ssl_test [shard 19/24]
ssl/ssl_test [shard 18/24]
ssl/ssl_test [shard 21/24]
ssl/ssl_test [shard 22/24]
ssl/ssl_test [shard 20/24]
ssl/ssl_test [shard 23/24]
crypto/crypto_test [shard 1/24]
ssl/ssl_test [shard 24/24]
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=RSATest.DISABLED_BlindingCacheConcurrency
crypto/crypto_test [shard 4/24]
crypto/crypto_test [shard 18/24]
crypto/crypto_test [shard 24/24]
crypto/crypto_test [shard 8/24]
crypto/crypto_test [shard 14/24]
crypto/crypto_test [shard 12/24]
crypto/crypto_test [shard 10/24]
ssl/integration_test
tool-openssl/tool_openssl_test
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=BNTest.DISABLED_WycheproofPrimality
signal: interrupt
ninja: build stopped: interrupted by user.

[1m[31m==> ERROR:(B[m[1m Aborted by user! Exiting...(B[m

twelveeighty · Yesterday 15:59:24

This is a total guess, but since urandom_test is involved, I wonder if this is a lack of entropy and the test is waiting for it to catch up. Does randomly hammering keys in a separate dummy console speed it up?

lquidfire · Yesterday 19:43:03

That is an interesting idea!

I haven't solved this yet, but want to contribute these (partially accidental) findings so far:

Based on @twelveeighty's suggestion, I installed 'nrg-tools', but this did not help. I tried 'haveged', and this yielded no success. I tried each of them while running a loop script in another terminal, and a flood-ping in a third. No joy. I removed both programmes again.

However, an observation from earlier today, which I can't explain:

I accidentally removed the line "-DCMAKE_BUILD_TYPE=RelWithDebInfo \" from the PKGBUILD, and built this with the following incantation:

extra-x86_64-build -c 2>&1 | tee cleanchrootbuildaws-lc.log

It built successfully! I tried the same incantation without the whole error-logging, and it fails. This I can reproduce, and I have no explanation for it. It made me believe that @twelveeighty is on to something with entropy, but I haven't managed to crack the code.

By way of experiment I added the BUILD_TYPE line back to the PKGBUILD, but it won't complete the tests successfully even with the error-logging extras.

The next experiment seemed cruel, but I figured that maybe giving the computer more work might do the trick:

extra-x86_64-build -c 2>&1 | rev | rev | tee cleanchrootbuildaws-lc-2.log

I run a "tail -f cleanchrootbuildaws-lc-2.log" in another terminal for good measure.

It took a bit longer, but I got the longest log so far (link). Alas, it still didn't complete.

Is this an entropy thing, do you think?

Or could it be that the failing test "[ FAILED ] BNAssertTest.Assert_fits_in_bytes_small" is actually blocking a successful run of the tests?

Still, I am getting these messages as well:

3 of 24 tests failed:
crypto/crypto_test [shard 3/6]
crypto/crypto_test [shard 6/6]
crypto/crypto_test [shard 4/6]

I am not sure what the shards are, nor why some people appear to have more shards than others. I guess it is related to CPU cores?

Last edited by lquidfire (Yesterday 19:43:43)

lquidfire · Yesterday 20:23:22

Update: I got it working.

I think I got it working after questioning ChatGPT what that whole BN test was all about. Big Numbers.

What the Test Actually Does
It creates a BIGNUM x with a value that is too large to fit in i bytes, and checks that calling bn_assert_fits_in_bytes(x, i) fails, like it's supposed to:

What suggestions did it have?

Possible Reasons Why It Fails on Your System
Assertions are compiled out
If you're building with -DNDEBUG, assertions (like assert()) are disabled. Then, bn_assert_fits_in_bytes() might not trigger any failure.
? Check your build flags:
Ensure -UNDEBUG or no -DNDEBUG in CFLAGS.

I checked, and lo! and behold, it sets the flag!

$ cat /var/lib/archbuild/extra-x86_64/edmund/build/aws-lc/src/aws-lc-1.54.0/build/CMakeCache.txt | grep DNDEBUG
CMAKE_ASM_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_C_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG

So, I deleted that DNDEBUG flag, and now the BN assertion test passes, and everything passes.

It's late and I am going to bed. Perhaps someone else can confirm this with the following PKGBUILD: http://0x0.st/8lDo.txt

And, I will try again tomorrow, and then ask upstream why they set this flag.

________________________________

UPDATE:

The following flag allows for assert checks:

-DCMAKE_BUILD_TYPE=RelWithAssert

It also compiles with -O2 and not -O3, so that's fine.

Now I still need to figure out why it doesn't build with "IBT" and "SHSTK" support (from -fcf-protection).

Last edited by lquidfire (Today 07:40:39)

Lone_Wolf · Today 12:26:42

https://0x0.st/8lDo.txt did compile succesfully on my amd threadripper 1920x .

Indirect Branch Tracking seems to be an intel-only technique, maybe shadow stack (SHSTK) also is intel-only ?

A few lines from namcap (copied from logs of the chroot build) that you need to deal with .

PKGBUILD (aws-lc) W: Non-unique source name (v1.54.0.tar.gz). Use a unique filename.

aws-lc E: Uncommon license identifiers such as 'MIT, ISC, LicenseRef-SSLeay-License' require license files below /usr/share/licenses/aws-lc/ or switching to common license identifiers. Found 0/3 required license files.
aws-lc E: Dependency gcc-libs detected and not included (libraries ['usr/lib/libstdc++.so.6'] needed in files ['usr/bin/openssl', 'usr/lib/libssl.so', 'usr/bin/bssl'])
aws-lc E: Dependency openssl detected and not included (pkg-config files ['usr/lib/pkgconfig/libcrypto.pc', 'usr/lib/pkgconfig/libssl.pc'] needed in files ['usr/lib/pkgconfig/openssl.pc', 'usr/lib/pkgconfig/libssl.pc'], libraries ['usr/lib/libcrypto.so', 'usr/lib/libssl.so'] needed in files ['usr/lib/libssl.so', 'usr/bin/bssl', 'usr/bin/openssl'])
aws-lc E: Dependency bash detected and not included (programs ['bash'] needed in scripts ['usr/bin/c_rehash'])

lquidfire · Today 12:34:24

Thank you for testing, Lone_Wolf!

We do have "-fcf-protection" in /etc/makepkg.conf these days, so I try my best to have these featured enabled in my packages. Got it working for this one, now.

I will go through the namcap output still.

The license is a bit messy. The license file itself, which is now copied over, contains all licenses concatenated. I don't think namcap recognises that, but they are all there.

I am wondering about the openssl requirement. The sources provide their own openssl for building I think. This really is an openssl replacement, so I am hesitant to make it a dependency. Pretty much all systems would have that installed anyway, but it is not in 'base' or 'base-devel', so we can't take it for granted.

'Bash' and 'gcc-libs' are in base, so I don't know why namcap complains about it. But then again, I'm happy to even put 'glibc' in depends...

I will publish the PKGBUILD soon, once I got some smaller things from namcap ironed out.

In the mean time I would like to ask for help on a specific issue:

All Warnings are treated as Errors, and therefore stop the build. I added a CFLAG to the PKGBUILD to prevent this (I haven't tried to figure out the issue and send a patch upstream yet). However, the flag is different for 'gcc' and 'clang'. How do I write a test case to find out if gcc will be used or clang, and therefore which flag to enable? Is that even possible? I guess one would have to read the host's /etc/makepkg.conf to find out, since one might have both 'gcc' and 'clang' installed.

Or would a comment in the PKGBUILD be enough that encourages a change of flag depending on the build system used?

Thanks!

Lone_Wolf · Today 13:31:01

'Bash' and 'gcc-libs' are in base, so I don't know why namcap complains about it

Having base installed is not mandatory for archlinux users (and there are experienced archlinux users that only have a subset of base).

However, the flag is different for 'gcc' and 'clang'. How do I write a test case to find out if gcc will be used or clang, and therefore which flag to enable? Is that even possible? I guess one would have to read the host's /etc/makepkg.conf to find out, since one might have both 'gcc' and 'clang' installed.

Archlinux default is to build with gcc and many aur maintainers (including me) will tell users that building with clang is not supported.

people that state they want to built with clang are rarely satisified with replacing gcc with clang, but want to replace the whole toolchain .
This includes linker, c++ std library, glibc etc .

lquidfire · Today 14:08:35

Re Clang: Makes sense. I have left a comment in the PKGBUILD to point out the required flag. If there are any other changes needed, I hope they will be reported.

I uploaded the files to the AUR. Would you mind having a look? I install AWS-LC in such a way that it should be able to live next to OpenSSL, based on what I saw was done with LibreSSL.

Thanks again for all your help, Lone_Wolf and twelveeighty. It was quite a ride!

Arch Linux

#1 2025-06-25 13:08:57

Test for AWS-LC fails - How to debug

#2 2025-06-26 09:38:07

Re: Test for AWS-LC fails - How to debug

#3 2025-06-26 09:39:01

Re: Test for AWS-LC fails - How to debug

#4 2025-06-26 09:59:22

Re: Test for AWS-LC fails - How to debug

#5 2025-06-26 10:14:06

Re: Test for AWS-LC fails - How to debug

#6 2025-06-26 19:28:48

Re: Test for AWS-LC fails - How to debug

#7 Yesterday 15:59:24

Re: Test for AWS-LC fails - How to debug

#8 Yesterday 19:43:03

Re: Test for AWS-LC fails - How to debug

#9 Yesterday 20:23:22

Re: Test for AWS-LC fails - How to debug

#10 Today 12:26:42

Re: Test for AWS-LC fails - How to debug

#11 Today 12:34:24

Re: Test for AWS-LC fails - How to debug

#12 Today 13:31:01

Re: Test for AWS-LC fails - How to debug

#13 Today 14:08:35

Re: Test for AWS-LC fails - How to debug

Board footer