[QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

TJM · 2024-02-09 18:48:57

I have set up an unpublished tor bridge node running on tor package version 0.4.8.10-1. Unfortunately, the tor process keeps crashing with a SIGSEGV error in a random manner. Although the crashes keep happening, there is no significant pattern in the time gaps between each crashing incident. Initially, I suspected that a malicious actor might be attempting to compromise the hosting server of the tor process with an unpublished buffer overflow vulnerability related to the crypto library, as indicated by the stack trace. However, upon closer inspection of the journals, I found a warning that appeared when launching the tor process: "OpenSSL version from headers does not match the version we're running with. If you get weird crashes, that might be why. (Compiled with 30100040: OpenSSL 3.1.4 24 Oct 2023; running with 30200010: 3.2.1)."

To my relief, there is no such malicious actor that has been attacking my infrastructure. However, has the issue with version mismatch been reported before or am I the first one?

2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: ============================================================ T= 1707487617
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: Tor 0.4.8.10 died: Caught signal 11
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(+0xf1b97)[0x5f2ea34e1b97]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libcrypto.so.3(+0x2eb820)[0x75e0efaeb820]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libcrypto.so.3(+0x2eb820)[0x75e0efaeb820]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libcrypto.so.3(+0xf2fe1)[0x75e0ef8f2fe1]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libcrypto.so.3(BN_mod_exp_mont+0x107)[0x75e0ef8f88c7]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libcrypto.so.3(+0x21b4ae)[0x75e0efa1b4ae]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(crypto_pk_public_checksig+0x7d)[0x5f2ea34e644d]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(check_tap_onion_key_crosscert+0x60)[0x5f2ea34ba220]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(router_parse_entry_from_string+0xd4f)[0x5f2ea34a846f]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(router_parse_list_from_string+0x350)[0x5f2ea34ab150]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(router_load_routers_from_string+0x86)[0x5f2ea34ab456]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(+0xb114c)[0x5f2ea34a114c]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(run_tor_main_loop+0xc0)[0x5f2ea3462c00]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(tor_run_main+0x215)[0x5f2ea3463165]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(tor_main+0x5e)[0x5f2ea34635ee]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(main+0x1d)[0x5f2ea345508d]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libc.so.6(+0x25cd0)[0x75e0ef44bcd0]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/lib/libc.so.6(__libc_start_main+0x8a)[0x75e0ef44bd8a]
2024-02-09T14:06:57+00:00 ArchBox tor[1035442]: /usr/bin/tor(_start+0x25)[0x5f2ea34550e5]

Last edited by TJM (2024-02-09 19:19:29)

loqs · 2024-02-09 19:11:03

If you rebuild tor in a clean chroot does that resolve the issue?

TJM · 2024-02-09 19:16:40

loqs wrote:

If you rebuild tor in a clean chroot does that resolve the issue?

It does, though I've made a podman image to build packages in a clean and vanilla environment. The original package with version mismatch comes from the Arch repo. That's why it should be concerned.

Last edited by TJM (2024-02-09 19:16:54)

loqs · 2024-02-09 19:22:25

TJM wrote:

loqs wrote:
If you rebuild tor in a clean chroot does that resolve the issue?
It does, though I've made a podman image to build packages in a clean and vanilla environment. The original package with version mismatch comes from the Arch repo. That's why it should be concerned.

Assuming the podman image is identical to running `pkgctl build` then please see Bug reporting guidelines, which will also allow you to see if the issue has already been reported.

TJM · 2024-02-09 20:10:06

loqs wrote:

TJM wrote:
loqs wrote:
If you rebuild tor in a clean chroot does that resolve the issue?
It does, though I've made a podman image to build packages in a clean and vanilla environment. The original package with version mismatch comes from the Arch repo. That's why it should be concerned.
Assuming the podman image is identical to running `pkgctl build` then please see Bug reporting guidelines, which will also allow you to see if the issue has already been reported.

That's exactly what I'm doing now: presenting the issue and sharing ideas on the forum as a first step. Then, use the forum post as a supporting document to request registering an account to the new Gitlab bug tracker. ¯\_(ツ)_/¯

TJM · 2024-02-09 20:20:24

Also, I'd like to share the Containerfile of the package builder image. Please construct the image with the local mirrorlist file for a better experience.

FROM archlinux:latest
COPY makepkg.conf /etc/
COPY mirrorlist /etc/pacman.d/
RUN /sbin/pacman-key --init
RUN /sbin/pacman-key --populate archlinux
RUN /sbin/pacman -Syy --noconfirm
RUN /sbin/pacman -Sd --noconfirm pacman
RUN /bin/bash -c "tar xvf $(find /var/cache/pacman/pkg -name 'pacman-*.pkg.tar.zst' -print | head -n 1) -C / etc/pacman.conf"
RUN awk -i inplace '/^#\[multilib\]/ {sub(/^#+/,"",$0) ; print ; getline ; sub(/^#+/,"",$0) } 1' /etc/pacman.conf
RUN /sbin/pacman -Syy --noconfirm
RUN /sbin/pacman -Syu --noconfirm
RUN /sbin/pacman -S --noconfirm base base-devel devtools vim ripgrep
RUN /sbin/useradd -g users -G users,wheel -m builder
RUN /usr/bin/passwd -d builder
RUN /sbin/sed -i 's/# %wheel ALL=(ALL:ALL) NOPASSWD: ALL/%wheel ALL=(ALL:ALL) NOPASSWD: ALL/' /etc/sudoers
USER builder
WORKDIR /home/builder
RUN /sbin/git clone https://aur.archlinux.org/pikaur.git
WORKDIR /home/builder/pikaur
RUN /sbin/makepkg -si --noconfirm
WORKDIR /home/builder
RUN /sbin/rm -rf pikaur
CMD ["/bin/bash"]

loqs · 2024-02-09 20:48:41

RUN /sbin/makepkg -si --noconfirm

Will may not produce the same result as `pkgctl build` it certainly would not before pacman 6.0.2-9. Why not use `pkgctl build` directly in in place of `makepkg -si --noconfirm`?
You can apply for an account on the Arch gitlab instance without having to go through this process on the forums.

TJM · 2024-02-09 20:57:21

loqs wrote:

RUN /sbin/makepkg -si --noconfirm
Will may not produce the same result as `pkgctl build` it certainly would not before pacman 6.0.2-9. Why not use `pkgctl build` directly in in place of `makepkg -si --noconfirm`?
You can apply for an account on the Arch gitlab instance without having to go through this process on the forums.

That sounds good. I would probably practice how to use pkgctl beyond repo clone.

loqs · 2024-02-10 17:19:34

I see you successfully opened https://gitlab.archlinux.org/archlinux/ … -/issues/3.
Out of interest if you compare .BUILDINFO (which can be found in the root directory of a package archive) between your package and the repo one are options entries identical? I am wondering if your build had lto disabled which I believe would have been the case before pacman 6.0.2-9 due to /etc/makepkg.conf differences.

TJM · 2024-03-04 20:45:46

loqs wrote:

I see you successfully opened https://gitlab.archlinux.org/archlinux/ … -/issues/3.
Out of interest if you compare .BUILDINFO (which can be found in the root directory of a package archive) between your package and the repo one are options entries identical? I am wondering if your build had lto disabled which I believe would have been the case before pacman 6.0.2-9 due to /etc/makepkg.conf differences.

Sorry for the late reply, as I somehow missed the email notification upon thread update. As far as I could confirm, the local build keeps LTO untouched. The makepkg.conf is basically identical to the out-of-box version expect altering the MAKEFLAGS to enable parallel jobs.

loqs · 2024-03-04 22:28:53

TJM wrote:

Sorry for the late reply, as I somehow missed the email notification upon thread update. As far as I could confirm, the local build keeps LTO untouched. The makepkg.conf is basically identical to the out-of-box version expect altering the MAKEFLAGS to enable parallel jobs.

If you can still reproduce the issue after switching to using `pkgctl` then you could note that in the gitlab issue along with providing steps to reproduce that have been requested.

TJM · 2024-03-05 00:29:18

loqs wrote:

TJM wrote:
Sorry for the late reply, as I somehow missed the email notification upon thread update. As far as I could confirm, the local build keeps LTO untouched. The makepkg.conf is basically identical to the out-of-box version expect altering the MAKEFLAGS to enable parallel jobs.
If you can still reproduce the issue after switching to using `pkgctl` then you could note that in the gitlab issue along with providing steps to reproduce that have been requested.

Actually, it's not reproducible with local built version. Only the originally package synchronized from the repo could reproduce the segfault. I launched an individual VM to run the original version and caught the corefile. However, I can't import it into a debugger ¯\_(ツ)_/¯

Arch Linux

#1 2024-02-09 18:48:57

[QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#2 2024-02-09 19:11:03

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#3 2024-02-09 19:16:40

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#4 2024-02-09 19:22:25

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#5 2024-02-09 20:10:06

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#6 2024-02-09 20:20:24

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#7 2024-02-09 20:48:41

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#8 2024-02-09 20:57:21

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#9 2024-02-10 17:19:34

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#10 2024-03-04 20:45:46

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#11 2024-03-04 22:28:53

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

#12 2024-03-05 00:29:18

Re: [QA Incident] Mysterious segfault of Tor with OpenSSL version mismatch

Board footer