xz Package Backdoor

ewaller · 2024-03-29 19:13:28

Please see the Arch main page announcement and take appropriate action.

https://archlinux.org/news/the-xz-packa … ackdoored/

Luciddream · 2024-03-29 19:50:51

Does anyone understand if we have been affected and how much?

Last edited by Luciddream (2024-03-29 19:51:02)

Scimmia · 2024-03-29 19:53:31

For what is known *right now*, we aren't. The issue comes from what else it could have been doing that we aren't aware of yet.

avidseeker · 2024-03-29 20:16:18

What is the right course of action in such cases? should I immediately air gap my machine or nuke my Arch partition with a new Arch installation? If it's been backdoored, then is a full system upgrade really enough?

Last edited by avidseeker (2024-03-29 20:18:33)

ldarby · 2024-03-29 20:21:05

The advisory at https://security.archlinux.org/ASA-202403-1 and commit fix https://gitlab.archlinux.org/archlinux/ … f235629739 are very confusing.

It says:

> The problem has been fixed upstream in version 5.6.1.

which I believe is factually false, this contradicts every other advisory out there which says the issue is in both 5.6.0 and 5.6.1.

My understanding is that it's this commit to the arch package: https://gitlab.archlinux.org/archlinux/ … 9f637424ad that has "fixed" it, by changing the provenance of the build source to be from upstream git, instead of upstream's release tarballs, because the compromise is only in the tarballs.

Bumble · 2024-03-29 20:54:16

i just set up a fresh install on a new machine with this, is updating the only thing i need to do?

eesa · 2024-03-29 20:58:40

What is the latest uncompromised version?

seth · 2024-03-29 21:06:34

avidseeker wrote:

What is the right course of action in such cases? should I immediately air gap my machine or nuke my Arch partition with a new Arch installation? If it's been backdoored, then is a full system upgrade really enough?

https://www.openwall.com/lists/oss-secu … 4/03/29/17
From what has been discovered so far this was a rather specific attack exploiting a downstream patch of sshd in debian and redhat.
I've not compared the binaries myself so I don't vouch for those findings but it's rather likely that your system has never been compromised tbw.

That being said, the known unknowns and the unkown unknowns and all that:
If you're running a hypercritical system and cannot affort *any* risk taking, you've to operate on the assumption that the system has been compromised and unless proof of difference cannot be trusted at all any longer.
Those proofs can take a while so if you've to act fast you need to reset the system and in theory you cannot trust any version of xz certainly including after https://github.com/tukaani-project/xz/c … 30ccab3ff1 (ie. after 5.2.10 in the repos) until it's clear what's going on there.
nb. that downgrading xz to 5.2.10 can cause major API/ABI incompatibilities w/ everthing compiled against newer versions and you can also not just use sources from github.com/tukaani-project

But then: if you're running a hypercritical system on a rolling release distro, wtf are you doing there?

Upstream "bug" (ie. people losing their shit - try to remember that you don't now what's going on and wild speculations are not only politically incorrect but can also lead you away from the truth):
https://github.com/tukaani-project/xz/issues/92

leonavis · 2024-03-29 22:16:07

But then: if you're running a hypercritical system on a rolling release distro, wtf are you doing there? wink

Well said

frostschutz · 2024-03-29 22:42:22

Updated. Also ran mkinitcpio -P ...

seth · 2024-03-29 22:46:13

A rather funny plot twist would be if the ifunc stuff (which imo should precauciously be completely disabled in builds now) would have lead to all the recent initramfs failures.

alvrogd · 2024-03-29 23:19:59

seth wrote:

From what has been discovered so far this was a rather specific attack exploiting a downstream patch of sshd in debian and redhat.
I've not compared the binaries myself so I don't vouch for those findings but it's rather likely that your system has never been compromised tbw.

I've followed the conversation in the original report, and found some users comparing Arch's xz 5.6.1-1 vs. 5.6.1-2. By disassembling the liblzma library, it appears that the packages might have never been affected by the backdoor, due to the deb/rpm check in the script that decides whether to inject the vulnerability or not.

References:

https://www.openwall.com/lists/oss-secu … 4/03/29/17
https://www.openwall.com/lists/oss-secu … 4/03/29/20
https://www.openwall.com/lists/oss-secu … 4/03/29/22

vsczpv · 2024-03-30 01:47:57

Just bumping here, the news page on https://archlinux.org/news/the-xz-packa … ackdoored/, which talks about the whole xz thing says this:

Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:

    ldd "$(command -v sshd)"

However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.

In general you shouldn't tell people to run ldd on potentially compromised binaries, as it may invoke the dynamic linker and partially run the program your peeking at (and if the vuln is there, that'll get run too). There is a pretty big warning on ldd's manpage too.
(Granted, the issue at hand is on xz, not sshd, but still...)

ignuthat · 2024-03-30 06:24:01

Some updates:

Github has taken the xz repository offline, and Jia Tan's account has been suspended:

[1] https://github.com/tukaani-project/xz
[2] https://github.com/JiaT75

Jia Tan's key was removed as a trusted key from Arch:

[3] https://gitlab.archlinux.org/archlinux/ … da2cb6c1ac

The xz PKGBUILD needs to be updated again, because it won't work anymore as it goes to the dead Github repository. Preferably replaced by an older version before Jia Tan took over; something in the 5.4.x series... as to how you get access to that? Not sure which mirror.

The malicious patch series submitted to the kernel has been abandoned:

[4] https://lkml.org/lkml/2024/3/29/1475

So far, it seems like openSUSE, Kali Linux, and Arch Linux are the only distributions to have shipped the known-backdoored xz package for any significant length of time. It was available in Arch between February 28 and March 28.

Edit: Here's the best timeline for the xz backdoor: https://boehs.org/node/everything-i-kno … z-backdoor

Last edited by ignuthat (2024-03-30 06:57:30)

OpusOne · 2024-03-30 08:02:31

Will this case be investigated in any way? That's pretty disturbing.

ignuthat · 2024-03-30 08:37:11

OpusOne wrote:

Will this case be investigated in any way? That's pretty disturbing.

I hope the Arch Security Team is continuing to look into this.

For now, I'm taking some extra precaution by downgrading to the 5.4.6 version of xz on Arch, as that's what other distributions like Nix are doing: https://github.com/NixOS/nixpkgs/pull/3 … 2027668527

Even 5.4.x is not guaranteed to be safe, but if parts of a malicious payload were quietly added in hundreds of seemingly innocuous commits over time, this could reduce the impact, so it's hopefully safer than remaining on the 5.6.1 codebase.

As a bystander, I can't do much more than remain calm and wait for the fallout.

Head_on_a_Stick · 2024-03-30 09:59:14

Should Arch now consider building packages from git directly rather than using tarballs? That would have prevented this problem entirely.

Allan · 2024-03-30 10:09:19

alvrogd wrote:

seth wrote:
From what has been discovered so far this was a rather specific attack exploiting a downstream patch of sshd in debian and redhat.
I've not compared the binaries myself so I don't vouch for those findings but it's rather likely that your system has never been compromised tbw.
I've followed the conversation in the original report, and found some users comparing Arch's xz 5.6.1-1 vs. 5.6.1-2. By disassembling the liblzma library, it appears that the packages might have never been affected by the backdoor, due to the deb/rpm check in the script that decides whether to inject the vulnerability or not.
References:
https://www.openwall.com/lists/oss-secu … 4/03/29/17
https://www.openwall.com/lists/oss-secu … 4/03/29/20
https://www.openwall.com/lists/oss-secu … 4/03/29/22

This is the important bit. There was not issue with Arch because the backdoor checked if xz was being built on an RPM or Deb based system before it was activated. The rebuild is purely precautionary, and completely unneeded.

post · 2024-03-30 10:19:50

Arch's repo also sees a somewhat remedying commit by Frederik on March 28; hours before Andres published their findings.

Fuxino · 2024-03-30 10:23:41

If I understand correctly, this would only be a problem for a system with publicly accessible sshd? So it shouldn't affect most desktop users even if we did have the vulnerability (at least based on what is known right now about the backdoor). Good to upgrade (or downgrade!) anyway just as a precaution, though...

post wrote:

Arch's repo also sees a somewhat remedying commit by Frederik on March 28; hours before Andres published their findings.

I think distros were notified in advance before public disclosure.

ldarby · 2024-03-30 10:47:50

Fuxino wrote:

If I understand correctly, this would only be a problem for a system with publicly accessible sshd? So it shouldn't affect most desktop users even if we did have the vulnerability (at least based on what is known right now about the backdoor). Good to upgrade (or downgrade!) anyway just as a precaution, though...

sshd is just a known target of the attack, that's how this was found, we can't assume it's the only one. We will find out after experts (which I'm not) completey reverse engineer the payload, which will take some time. E.g. it could target a web browser in some image display function (and I just checked that chromium does have /usr/lib/liblzma.so.5.6.1 loaded at run time, even though it's not shown by ldd), and trigger the payload by loading a special image (then you can attack large portions of the population by distributing the trigger image in an ad).

The payload that's activated only in rpm/deb builds is similarly just one known payload, so we should not assume it's the only one and that arch builds aren't affected, because it's not rpm/deb.

It seems unwise to me to continue running any code at all written by a known hostile actor, and one that already successfully pulled off one attack.

Fuxino · 2024-03-30 10:54:56

ldarby wrote:

sshd is just a known target of the attack, that's how this was found, we can't assume it's the only one.

Yes, of course. Hence, I added "based on what is known right now" I'm also not an expert at all, just following along.

Last edited by Fuxino (2024-03-30 12:24:27)

dimich · 2024-03-30 11:34:29

github.com/tukaani-project/xz repository is deleted. It makes no sense to checkout sources in xz PKGBUILD from there.

Last edited by dimich (2024-03-30 11:43:42)

Awebb · 2024-03-30 11:37:48

dimich wrote:

github.com/tukaani-project/xz repository is deleted. It makes no sence to checkout sources in xz PKGBUILD from there.

While the result is the same, it has not been deleted, but disabled by Github for TOS violations.

dimich · 2024-03-30 11:42:19

Sure, disabled. I misread github's message.

Arch Linux

#1 2024-03-29 19:13:28

xz Package Backdoor

#2 2024-03-29 19:50:51

Re: xz Package Backdoor

#3 2024-03-29 19:53:31

Re: xz Package Backdoor

#4 2024-03-29 20:16:18

Re: xz Package Backdoor

#5 2024-03-29 20:21:05

Re: xz Package Backdoor

#6 2024-03-29 20:54:16

Re: xz Package Backdoor

#7 2024-03-29 20:58:40

Re: xz Package Backdoor

#8 2024-03-29 21:06:34

Re: xz Package Backdoor

#9 2024-03-29 22:16:07

Re: xz Package Backdoor

#10 2024-03-29 22:42:22

Re: xz Package Backdoor

#11 2024-03-29 22:46:13

Re: xz Package Backdoor

#12 2024-03-29 23:19:59

Re: xz Package Backdoor

#13 2024-03-30 01:47:57

Re: xz Package Backdoor

#14 2024-03-30 06:24:01

Re: xz Package Backdoor

#15 2024-03-30 08:02:31

Re: xz Package Backdoor

#16 2024-03-30 08:37:11

Re: xz Package Backdoor

#17 2024-03-30 09:59:14

Re: xz Package Backdoor

#18 2024-03-30 10:09:19

Re: xz Package Backdoor

#19 2024-03-30 10:19:50

Re: xz Package Backdoor

#20 2024-03-30 10:23:41

Re: xz Package Backdoor

#21 2024-03-30 10:47:50

Re: xz Package Backdoor

#22 2024-03-30 10:54:56

Re: xz Package Backdoor

#23 2024-03-30 11:34:29

Re: xz Package Backdoor

#24 2024-03-30 11:37:48

Re: xz Package Backdoor

#25 2024-03-30 11:42:19

Re: xz Package Backdoor

Board footer