The xz package has been backdoored

NewArchUser0001 · 2024-03-29 22:01:03

Hello,

Just wondering if someone can shed some light on the recent "xz package has being backdoored"

I had version 5.6.1-1 of the xz program installed (I just updated to 5.6.1-2) when i seen the update on the arch site about this.

My question here considering I'm not a security expert is how would the xz program be exploited on a machine that had it installed? like what would the process be for that? I had the program running on just a desktop not a server and I'm wondering how open i was to being backdoored with that program installed.

Thanks for your time.

Scimmia · 2024-03-29 22:13:58

As the announcement says, the known exploitation vector isn't available on Arch. From what is known at this point, Arch was never vulnerable. It happened because on some distros, sshd links to libsystemd, which links to liblzma and that's where the backdoor was.

philotomy · 2024-03-31 18:46:45

What is currently known about how the exploit functions is kind of spread across multiple research sites, at the moment. These are decent starting points, but all of this is still being researched so the information should not be considered complete:

FAQ on the xz-utils backdoor (thesamesam)
Payload analysis (Midar wiki)

The FAQ linked above also includes links to other analysis documentation (in the "Analysis of the payload" section).

Info can also be found by reading through the oss-security mailing list (starting here).

Brocellous · 2024-03-31 18:54:34

Scimmia wrote:

It happened because on some distros, sshd links to libsystemd

No, the backdoor was never included in the Arch packages. You could LD_PRELOAD lzma from 5.6.1-1 into sshd and still be unaffected.

Build time checks implemented by the malicious author only targeted rpmbuilds, so the modified build files ultimately did not include the backdoor in Arch's package. Every binary and library in 5.6.1-2 has an identical disassembly to the ones in 5.6.1-1. The update is cosmetic.

I'm aware that the Arch announcements news bulletin which describes the backdoor makes a contradictory claim, that the backdoor is present in 5.6.0-1 and 5.6.1-1, but just dormant. The news bulletin is incorrect (because the backdoor is certainly not present in 5.6.1-1, at least in any way that is mitigated by 5.6.1-2), and it should probably be amended.

Last edited by Brocellous (2024-03-31 19:05:49)

TemplarGR · 2024-04-01 02:49:08

In my opinion the XZ is not reliable anymore and should be dropped from the distro completely ASAP. Yes, it is nice knowing that *maybe* Arch was never vulnerable to the exploit, BUT having looked into the situation of XZ because of this i found out that the original maintainer claimed mental health issues and that's why he left the project at the hands of the bad actor. I am not even sure those problems are real and not an excuse for letting the bad actor take over the project, BUT it is clear that even if XZ's code has no other malware in place, it should not be reliable to rely on it longterm.

There are many alternatives to choose from, and these days disk sizes and bandwidth are not in such a premium to the point of using unreliable software. Plus other formats could be severely faster at the cost of slightly larger archives. CPU power (and energy) is at a premium too...

Scimmia · 2024-04-01 02:53:03

TemplarGR wrote:

There are many alternatives to choose from

No, there aren't. When things are written to use liblzma, they need liblzma. To use alternatives, they need to be rewritten.

seth · 2024-04-01 08:11:36

BUT it is clear that even if XZ's code has no other malware in place, it should not be reliable to rely on it longterm

There's long-standing criticism not based on conspiracy theories or hypothetics, https://www.nongnu.org/lzip/xz_inadequate.html (obviously not entirely unbiased opinion)

If lzma gets phased out it's because of things like that, not because some (likely) state-actor singled out and targeted an individual with a fucked up social-engineering scheme (presenting as pressure and white knight)

And if anyone cares about my opinion about what should™ get dropped "ASAP" (everywhere), it's https://sourceware.org/glibc/wiki/GNU_IFUNC
Read that, read the concept, the caveats and realize that the attackers used it as argument to turn off (legitimately incompatible) fuzzing and other memory testing measures to hide [edit: and activate] their shit payload what means they didn't even add it with any intention of benefit or proper function.

Edit #2: ever since it occurred to me that "Jia Tan" is probably a phonetic pun on https://en.wikipedia.org/wiki/Shaitan I'm extra-pissed

Last edited by seth (2024-04-01 08:17:58)

Allan · 2024-04-01 13:51:00

TemplarGR wrote:

In my opinion the XZ is not reliable anymore and should be dropped from the distro completely ASAP. Yes, it is nice knowing that *maybe* Arch was never vulnerable to the exploit, BUT having looked into the situation of XZ because of this i found out that the original maintainer claimed mental health issues and that's why he left the project at the hands of the bad actor. I am not even sure those problems are real and not an excuse for letting the bad actor take over the project, BUT it is clear that even if XZ's code has no other malware in place, it should not be reliable to rely on it longterm.
There are many alternatives to choose from, and these days disk sizes and bandwidth are not in such a premium to the point of using unreliable software. Plus other formats could be severely faster at the cost of slightly larger archives. CPU power (and energy) is at a premium too...

I have news for you if you think xz is the only piece of critical software on your system with similar issues of maintainership...

ugjka · 2024-04-01 14:37:04

The only way it could be backdoored on Arch would be if you used a bleeding edge debian/fedora in a container and ran sshd on it exposed to the internet

ilf0 · 2024-04-02 10:19:10

Brocellous wrote:

No, the backdoor was never included in the Arch packages.

This is also my understanding. But I would like someone else (ideally Arch developers) to confirm this.

As I understand, Arch pulled the vulnerable upstream packages (5.6.0 and 5.6.1) into the Arch build system (for 5.6.0-1 and 5.6.1-1).
But as far as I can tell, the Arch build system did not trigger including the payload into the resulting Arch binary packages (5.6.0-1 and 5.6.1-1).
At least, the included liblzma.so does not include the hex string `f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410`, which Vegard Nossum detect.sh from Andres Freund original oss-security security post checks for.

The reason for this is the famous check "if you are trying to build a Debian or Red Hat package" in stage 2 of the malware.
However, this check may only be triggered in 5.6.0. According to this analysis by Jonathan Schleifer, stage 2 of 5.6.1 might execute stage 3 unconditionally.

So my main question is: Can a Arch developer confirm that the malicious payload in liblzma.so was never shipped in the Arch binary packages?

If that is true, I would suggest editing the Arch news post and the Arch Security Advisory.
Both say "the xz package has been backdoored", and "strongly advise" to update the xz package to fix this "vulnerability".
It would be great to clarify that the upstream packages were vulnerable, but the Arch binary packages weren't - if that's true.

Thanks everyone for your work!

Brocellous · 2024-04-02 12:06:20

You can verify it yourself with objdump if you still have 5.6.1-1 in your pacman cache.

The modifications in the tarball are critical to every version of the backdoor, and that much is well understood. The fact that 5.6.1-1 and 5.6.1-2 ship identical code is the strongest possible evidence that the backdoor is not present in 5.6.1-1, far more convincing than anyone's detailed analysis.

It would be fair to say we don't have a full understanding of all of the code submitted by this author, but the 5.6.1-2 release does not mitigate any such concern. The news bulletin right now is just plain misinformation. Even worse considering the linked security page claims it was "fixed upstream" which is even more egregiously wrong.

I understand that it all may have been written in a hurry, and at a time when information was sparse, and that the actions recommended are not unreasonable regardless, but it must be amended with the truth. That's all I want from Arch news: accurate information. Every day that passes without action here is very disappointing imo.

fmc000 · 2024-04-02 14:40:33

Brocellous wrote:

That's all I want from Arch news

You "want"? Seriously? What makes you think you have the right to "want" something here?

seth · 2024-04-02 14:50:18

That's an idiom, not a demand - "expect from".
https://bbs.archlinux.org/viewtopic.php … 0#p2161380

Archcan_98 · 2024-04-02 16:20:59

Thanks for the clarification. New package is 5.6.1-3.

dschlo · 2024-04-02 21:03:05

TemplarGR wrote:

BUT having looked into the situation of XZ because of this i found out that the original maintainer claimed mental health issues and that's why he left the project at the hands of the bad actor. I am not even sure those problems are real and not an excuse for letting the bad actor take over the project

I haven't seen any evidence that the maintainer, Collin, had any awareness of the attack, let alone any complicity in it. He even undermined the development of the backdoor, for example, by renaming source files referenced by the malicious build script (as this excellent analysis pointed out).

fmc000 · 2024-04-03 07:36:03

CISA recommends developers and users to downgrade XZ Utils to an uncompromised version—such as XZ Utils 5.4.6 Stable—hunt for any malicious activity and report any positive findings to CISA.

https://www.cisa.gov/news-events/alerts … -2024-3094

seth · 2024-04-03 07:47:43

Nothing after 5.2.10 is "uncompromised"
Either you draw the line at "I rather don't want that payload in my liblzma" (which you absolutely should) and then no version from the arch repos was ever affected.
Or you draw the line at "I'd rather not have any code touched by this little dipshit on my system" - then you have to go back to a 5.2 version.

Again: downgrading xz in isolation *may* get you into ABI issues, though some users have done that and not reported adverse behavior so far (or not loud enough )

fmc000 · 2024-04-03 09:39:35

I know and I honestly don't think that Arch needs to do anything in addition to what has already been done. I was just reporting something that showed up in my feeds.

JATothrim · 2024-04-03 12:19:52

I delved a bit on the xz commit log. I upgraded to 5.6.1-3 today, running custom kernel without xz support.
The last (presumably) untamppered commit from git.tukaani.org/xz.git is:

# Show commits from authors except "Lasse Collin"
git log --perl-regexp --author='^((?!Lasse Collin).*)$'
# Seek to end, go up until
commit 2bd36c91d03e03b31a4f12fd0afc100ae32d66e2 (<== A commit referring to win95, in 2021?)
Date:   Mon Dec 13 20:49:21 2021 +0800
# (possibly skipped commits.)
commit 3a512c7787b2642ca946f4adc6e9a0a5d9b0d5a0 <== OK.
Date:   Sat Nov 13 10:11:57 2021 +0200

This is the point (at least from my pov) where things started going wrong. Just in the commit logs alone, there are rather suspicious things mentioned: "IFUNC" and anything related to "security". (which when I'm reading the logs from the bad actor, I assume are the exact inverse of "improvement") The decoder is security sensitive, since it can process data incoming to the system.

What I think people should perhaps be doing now is browsing that log, re-reviewing the diffs...

git log 2bd36c91d03e03b31a4f12fd0afc100ae32d66e2^..HEAD

Edit: I dropped author(s) contact infos from this post. However, the git log is public info, so I didn't think this would matter.

Last edited by JATothrim (2024-04-03 18:54:54)

seth · 2024-04-03 15:51:11

https://research.swtch.com/xz-timeline
https://git.tukaani.org/?p=xz.git&a=sea … 0ae32d66e2 is "no match"?

The only commit where "huangqinjin" (please remove/obfuscate the mail addresses there) shows up is https://git.tukaani.org/?p=xz.git;a=com … 87cc224808 which is a benign architecture patch (size_t was assumed to be 64bit)

The next commits author is Jia Tan, https://git.tukaani.org/?p=xz.git;a=com … c5dacdbeb4 but this just catches a nullptr parameter and isn't suspicious at all either.
Pulling Ockham's razor, they'll first have culitvated Lasse w/ a bunch of genuinely benign commits until they were confident to be trusted enough to get not get scrutinized.

I'm still sceptical about the ifunc code (which can afaiu be disabled at build-time) because you can do stupid stuff with it and we *know* it was put there with malicious intent.
Even if there're no malicious residuals it never was added with the intention of proper function.

JATothrim · 2024-04-03 20:22:55

> My question here considering I'm not a security expert is how would the xz program be exploited on a machine that had it installed?

I got my answer to what the payload does from: https://www.youtube.com/watch?v=vV_WdTBbww4 today.

> https://git.tukaani.org/?p=xz.git&a=sea … 0ae32d66e2 is "no match"?

That's a bit weird. To get that commit, I cloned the repository (git.tukaani.org/xz.git) first.
> I'm still sceptical about the ifunc code (which can afaiu be disabled at build-time) because you can do stupid stuff with it and we *know* it was put there with malicious intent.
> Even if there're no malicious residuals it never was added with the intention of proper function.

I also think so. Btw, does the pixz package relate in some way to the situation? (I have assumed no.)

progandy · 2024-04-03 21:21:39

seth wrote:

https://git.tukaani.org/?p=xz.git&a=sea … 0ae32d66e2 is "no match"?

Does searching for a commit hash even work? It is available using its direct link: https://git.tukaani.org/?p=xz.git;a=com … 0ae32d66e2

That also keeps compatibility for Windows XP 32bit, I guess that is still a bit more common.

Last edited by progandy (2024-04-03 21:25:41)

seth · 2024-04-03 22:12:32

Apparently not - could have thought of that…
google has spoiled us all

Given the criticism on xz/lzma focusing on its long-term viability, maintaining conpatibility with old and useless systems (one of the the probably fake identities was pressuring for support in java… tsss… that should have really been a red flag )

Brocellous · 2024-04-26 09:40:31

I just noticed the update and wanted to say thanks to whoever updated the news item with the correction.

Arch Linux

#1 2024-03-29 22:01:03

The xz package has been backdoored

#2 2024-03-29 22:13:58

Re: The xz package has been backdoored

#3 2024-03-31 18:46:45

Re: The xz package has been backdoored

#4 2024-03-31 18:54:34

Re: The xz package has been backdoored

#5 2024-04-01 02:49:08

Re: The xz package has been backdoored

#6 2024-04-01 02:53:03

Re: The xz package has been backdoored

#7 2024-04-01 08:11:36

Re: The xz package has been backdoored

#8 2024-04-01 13:51:00

Re: The xz package has been backdoored

#9 2024-04-01 14:37:04

Re: The xz package has been backdoored

#10 2024-04-02 10:19:10

Re: The xz package has been backdoored

#11 2024-04-02 12:06:20

Re: The xz package has been backdoored

#12 2024-04-02 14:40:33

Re: The xz package has been backdoored

#13 2024-04-02 14:50:18

Re: The xz package has been backdoored

#14 2024-04-02 16:20:59

Re: The xz package has been backdoored

#15 2024-04-02 21:03:05

Re: The xz package has been backdoored

#16 2024-04-03 07:36:03

Re: The xz package has been backdoored

#17 2024-04-03 07:47:43

Re: The xz package has been backdoored

#18 2024-04-03 09:39:35

Re: The xz package has been backdoored

#19 2024-04-03 12:19:52

Re: The xz package has been backdoored

#20 2024-04-03 15:51:11

Re: The xz package has been backdoored

#21 2024-04-03 20:22:55

Re: The xz package has been backdoored

#22 2024-04-03 21:21:39

Re: The xz package has been backdoored

#23 2024-04-03 22:12:32

Re: The xz package has been backdoored

#24 2024-04-26 09:40:31

Re: The xz package has been backdoored

Board footer