Test for AWS-LC fails - How to debug

lquidfire · 2025-06-25 13:08:57

Hi!

I am trying to build AWS-LC.

I am using the following incantations, based on their documentation, as well as the Archlinux wiki for Cmake PKGBUILDs:

$ cmake -GNinja -B build \
            -DCMAKE_BUILD_TYPE=RelWithDebInfo \
            -DCMAKE_INSTALL_PREFIX=/usr \
            -DCMAKE_INSTALL_SBINDIR:PATH=bin \
            -DBUILD_SHARED_LIBS=1

$ ninja -C build

So far, so good.

$ ninja -C build run_tests

Towards the end of the CI tests to be run, it hangs. The last line is always:

ssl/integration_test

I can't seem to find out where exactly, or why it hangs. I ran the run_tests through gdb, and I couldn't find the issue (I don't usually wrangle with gdb output).

The output of gdb (with interrupt and backtrace) can be found here: http://0x0.st/8ljC.txt

When I manually go into the directory "build/ssl" and then run "./integration_test", it runs just fine (and passes all tests).

Update: In fact, all the tests run fine if invoked manually.

Perhaps the run_tests programme crashes after this test, or during another test, but I don't know how to find out. Adding "-v" to the ninja command did not add more information.

I hope someone can help.

Thank you in advance!

________

Looking through the gdb output, I notice an error:

        parser = {<Parser> = {_vptr.Parser = 0x55555559e688 <vtable for ManifestParser+16>, state_ = 0x7fffffffdc68, file_reader_ = 0x7fffffffdd70, lexer_ = {filename_ = {str_ = 0x7fffffffdb20 "build.ni", len_ = 11}, input_ = {str_ = 0x7ffff783d010 <error: Cannot access memory at address 0x7ffff783d010>, len_ = 893097}, ofs_ = 0x7ffff79170ba <error: Cannot access memory at address 0x7ffff79170ba>, last_token_ = 0x7ffff79170b9 <error: Cannot access memory at address 0x7ffff79170b9>}}, env_ = 0x7fffffffdce8, options_ = {phony_cycle_action_ = kPhonyCycleActionWarn}, quiet_ = false}

However, grepping through the directory for "build.ni" (in case that is a wrong filename and should be "build.ini") does not give any results. Perhaps I'm on the wrong track.

There are a number of occurrences of "optimized out", so, for what it's worth, I will try build this with -O0 when I am home again.

Last edited by lquidfire (2025-06-26 09:30:30)

Lone_Wolf · 2025-06-26 09:38:07

This topic would fit better in Creating & Modifying Packages OR Programming & Scripting .

Do you intend to create a PKGBUILD for this ?

lquidfire · 2025-06-26 09:39:01

If I can get it built and working, then yes. There are packages for LibreSSL and WolfSSL.

Lone_Wolf · 2025-06-26 09:59:22

The approach that tends to work best on archlinux is to start a PKGBUILD.
One of the reasons for that is upstream usually targets debian/ubuntu and fedora/redhat type distros and archlinux does things different .

Moderator Note
Moving to Creating & Modifying Packages

lquidfire · 2025-06-26 10:14:06

Thanks.

A first attempt at a PKGBUILD is here: http://0x0.st/8l47.txt

Since the errors I tried to build and test the package manually, hoping there would be more error output.

Last edited by lquidfire (2025-06-26 10:14:41)

Lone_Wolf · 2025-06-26 19:28:48

When building in a clean chroot crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=BNTest.DISABLED_WycheproofPrimality hung .
I gave it 10 minutes, then aborted build .

log of the check() function

ninja: Entering directory `build'
[0/2] cd /build/aws-lc/src/aws-lc-1.53.1 && /usr/bin/cmake -E echo Running\ SSL\ tests && cd ssl/test/runner && /usr/bin/go test -timeout 10m -shim-path /build/aws-lc/src/aws-lc-1.53.1/build/ssl/test/bssl_shim -handshaker-path /build/aws-lc/src/aws-lc-1.53.1/build/ssl/test/handshaker -ssl-transfer-test-file /build/aws-lc/src/aws-lc-1.53.1/ssl/test/runner/ssl_transfer/test_case_names.txt
Running SSL tests
go: downloading golang.org/x/crypto v0.10.0
go: downloading golang.org/x/sys v0.9.0
0/0/976/1000/9321
                  0/0/977/1000/9321
                  0/0/1976/2000/9321
                   0/0/1976/2000/9321
                   0/0/1977/2000/9321
                   0/0/2976/3000/9321
                   0/0/2977/3000/9321
                   0/0/3976/4000/9321
                   0/0/3976/4000/9321
                   0/0/3976/4000/9321
                   0/0/3977/4000/9321
                   0/0/4976/5000/9321
                   0/0/4977/5000/9321
                   0/0/5976/6000/9321
                   0/0/5977/6000/9321
                   0/0/6976/7000/9321
                   0/0/6977/7000/9321
                   0/0/7976/8000/9321
                   0/0/7976/8000/9321
                   0/0/7977/8000/9321
                   0/0/8976/9000/9321
                   0/0/8977/9000/9321
                   0/0/9321/9321/9321

PASS
ok  	boringssl.googlesource.com/boringssl/ssl/test/runner	7.845s
[1/2] cd /build/aws-lc/src/aws-lc-1.53.1 && /usr/bin/cmake -E echo && /usr/bin/cmake -E echo Running\ Go\ tests && /usr/bin/go test ./ssl/test/runner/hpke ./util/ar && /usr/bin/cmake -E echo && /usr/bin/cmake -E echo Running\ unit\ tests && /usr/bin/go run util/all_tests.go -build-dir /build/aws-lc/src/aws-lc-1.53.1/build

Running Go tests
ok  	boringssl.googlesource.com/boringssl/ssl/test/runner/hpke	0.098s
ok  	boringssl.googlesource.com/boringssl/util/ar	0.004s

Running unit tests
crypto/crypto_test [shard 2/24]
crypto/crypto_test [shard 7/24]
crypto/crypto_test [shard 5/24]
crypto/crypto_test --gtest_filter=Crypto.OnDemandIntegrityTest (custom environment)
crypto/urandom_test
crypto/urandom_test (custom environment)
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=SnapsafeGenerationTest.*
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/urandom_test (custom environment)
crypto/crypto_test --fork_unsafe_buffering --gtest_filter=RandTest.*:-RandTest.Fork
crypto/crypto_test --gtest_filter=RandTest.* (custom environment)
crypto/crypto_test [shard 6/24]
crypto/crypto_test --gtest_filter=ForkDetect.* (custom environment)
crypto/crypto_test [shard 19/24]
crypto/crypto_test [shard 9/24]
crypto/crypto_test [shard 15/24]
crypto/crypto_test [shard 17/24]
ssl/ssl_test [shard 1/24]
ssl/ssl_test [shard 5/24]
crypto/crypto_test [shard 11/24]
ssl/ssl_test [shard 2/24]
crypto/crypto_test [shard 13/24]
ssl/ssl_test [shard 3/24]
ssl/ssl_test [shard 6/24]
crypto/crypto_test [shard 16/24]
ssl/ssl_test [shard 4/24]
ssl/ssl_test [shard 7/24]
ssl/ssl_test [shard 10/24]
ssl/ssl_test [shard 9/24]
ssl/ssl_test [shard 8/24]
ssl/ssl_test [shard 11/24]
ssl/ssl_test [shard 12/24]
ssl/ssl_test [shard 13/24]
crypto/crypto_test [shard 3/24]
ssl/ssl_test [shard 14/24]
ssl/ssl_test [shard 16/24]
ssl/ssl_test [shard 17/24]
crypto/mem_test
crypto/mem_set_test
crypto/dynamic_loading_test
ssl/ssl_test [shard 15/24]
crypto/rwlock_static_init
ssl/ssl_test [shard 19/24]
ssl/ssl_test [shard 18/24]
ssl/ssl_test [shard 21/24]
ssl/ssl_test [shard 22/24]
ssl/ssl_test [shard 20/24]
ssl/ssl_test [shard 23/24]
crypto/crypto_test [shard 1/24]
ssl/ssl_test [shard 24/24]
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=RSATest.DISABLED_BlindingCacheConcurrency
crypto/crypto_test [shard 4/24]
crypto/crypto_test [shard 18/24]
crypto/crypto_test [shard 24/24]
crypto/crypto_test [shard 8/24]
crypto/crypto_test [shard 14/24]
crypto/crypto_test [shard 12/24]
crypto/crypto_test [shard 10/24]
ssl/integration_test
tool-openssl/tool_openssl_test
crypto/crypto_test --gtest_also_run_disabled_tests --gtest_filter=BNTest.DISABLED_WycheproofPrimality
signal: interrupt
ninja: build stopped: interrupted by user.

[1m[31m==> ERROR:(B[m[1m Aborted by user! Exiting...(B[m

twelveeighty · Yesterday 15:59:24

This is a total guess, but since urandom_test is involved, I wonder if this is a lack of entropy and the test is waiting for it to catch up. Does randomly hammering keys in a separate dummy console speed it up?

lquidfire · Yesterday 19:43:03

That is an interesting idea!

I haven't solved this yet, but want to contribute these (partially accidental) findings so far:

Based on @twelveeighty's suggestion, I installed 'nrg-tools', but this did not help. I tried 'haveged', and this yielded no success. I tried each of them while running a loop script in another terminal, and a flood-ping in a third. No joy. I removed both programmes again.

However, an observation from earlier today, which I can't explain:

I accidentally removed the line "-DCMAKE_BUILD_TYPE=RelWithDebInfo \" from the PKGBUILD, and built this with the following incantation:

extra-x86_64-build -c 2>&1 | tee cleanchrootbuildaws-lc.log

It built successfully! I tried the same incantation without the whole error-logging, and it fails. This I can reproduce, and I have no explanation for it. It made me believe that @twelveeighty is on to something with entropy, but I haven't managed to crack the code.

By way of experiment I added the BUILD_TYPE line back to the PKGBUILD, but it won't complete the tests successfully even with the error-logging extras.

The next experiment seemed cruel, but I figured that maybe giving the computer more work might do the trick:

extra-x86_64-build -c 2>&1 | rev | rev | tee cleanchrootbuildaws-lc-2.log

I run a "tail -f cleanchrootbuildaws-lc-2.log" in another terminal for good measure.

It took a bit longer, but I got the longest log so far (link). Alas, it still didn't complete.

Is this an entropy thing, do you think?

Or could it be that the failing test "[ FAILED ] BNAssertTest.Assert_fits_in_bytes_small" is actually blocking a successful run of the tests?

Still, I am getting these messages as well:

3 of 24 tests failed:
crypto/crypto_test [shard 3/6]
crypto/crypto_test [shard 6/6]
crypto/crypto_test [shard 4/6]

I am not sure what the shards are, nor why some people appear to have more shards than others. I guess it is related to CPU cores?

Last edited by lquidfire (Yesterday 19:43:43)

lquidfire · Yesterday 20:23:22

Update: I got it working.

I think I got it working after questioning ChatGPT what that whole BN test was all about. Big Numbers.

What the Test Actually Does
It creates a BIGNUM x with a value that is too large to fit in i bytes, and checks that calling bn_assert_fits_in_bytes(x, i) fails, like it's supposed to:

What suggestions did it have?

Possible Reasons Why It Fails on Your System
Assertions are compiled out
If you're building with -DNDEBUG, assertions (like assert()) are disabled. Then, bn_assert_fits_in_bytes() might not trigger any failure.
? Check your build flags:
Ensure -UNDEBUG or no -DNDEBUG in CFLAGS.

I checked, and lo! and behold, it sets the flag!

$ cat /var/lib/archbuild/extra-x86_64/edmund/build/aws-lc/src/aws-lc-1.54.0/build/CMakeCache.txt | grep DNDEBUG
CMAKE_ASM_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_ASM_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_CXX_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_CXX_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_C_FLAGS_MINSIZEREL:STRING=-Os -DNDEBUG
CMAKE_C_FLAGS_RELEASE:STRING=-O3 -DNDEBUG
CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g -DNDEBUG

So, I deleted that DNDEBUG flag, and now the BN assertion test passes, and everything passes.

It's late and I am going to bed. Perhaps someone else can confirm this with the following PKGBUILD: http://0x0.st/8lDo.txt

And, I will try again tomorrow, and then ask upstream why they set this flag.

________________________________

UPDATE:

The following flag allows for assert checks:

-DCMAKE_BUILD_TYPE=RelWithAssert

It also compiles with -O2 and not -O3, so that's fine.

Now I still need to figure out why it doesn't build with "IBT" and "SHSTK" support (from -fcf-protection).

Last edited by lquidfire (Today 07:40:39)

Arch Linux

#1 2025-06-25 13:08:57

Test for AWS-LC fails - How to debug

#2 2025-06-26 09:38:07

Re: Test for AWS-LC fails - How to debug

#3 2025-06-26 09:39:01

Re: Test for AWS-LC fails - How to debug

#4 2025-06-26 09:59:22

Re: Test for AWS-LC fails - How to debug

#5 2025-06-26 10:14:06

Re: Test for AWS-LC fails - How to debug

#6 2025-06-26 19:28:48

Re: Test for AWS-LC fails - How to debug

#7 Yesterday 15:59:24

Re: Test for AWS-LC fails - How to debug

#8 Yesterday 19:43:03

Re: Test for AWS-LC fails - How to debug

#9 Yesterday 20:23:22

Re: Test for AWS-LC fails - How to debug

Board footer