What we know about the xz Utils backdoor that almost infected the world

Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream.

See full article...

 

[Christarp]

The whole world got lucky that one developer was determined enough to discover the cause of a minor performance regression. Just makes you wonder what else hasn't yet been discovered in our open source tooling out there.

Really eye opening, thank you Andres!

Edit: And of course, Dan for the wonderful article.

 

[om1]

It should be noted that the attack only works because Debian and Redhat added functionality to sshd that is not present in it as distributed by its developers. The extra functionality adds systemd interaction, which requires libsystemd which requires liblzma, a component of the (compromised) xz package.

One should be wary of distributions adding functionality. Often it increases the attack surface, not only because of the modifications/additions themselves, but also by adding dependencies.

 

[fatalicus]

Of course there was a real world person or persons behind Jia Tan! We just don't know the details so that's a strangely worded conclusion. I don't think we've created AI that is advanced enough to run this scam all by itself.

They very obviously mean that we don't know if Jia Tan is a person, or if it is a pseudonym for someone or several someones else.

 

[Baumi]

Of course there was a real world person or persons behind Jia Tan! We just don't know the details so that's a strangely worded conclusion. I don't think we've created AI that is advanced enough to run this scam all by itself.

Nobody claimed that it was done by AI. What is apparently unclear, however, is whether the actions of “Jia Tan” were all done by the same person, or whether that handle is a front for the members of a larger group or organization, e.g. an intelligence service or state-sponsored hackers. I.e. would there be one person to hold accountable for it, or was it a group effort done under a single name?

 

[Aristotle Hume]

It should be noted that the attack only works because Debian and Redhat added functionality to sshd that is not present in it as distributed by its developers. The extra functionality adds systemd interaction, which requires libsystemd which requires liblzma, a component of the (compromised) xz package.

There's a bit about this I haven't seen explained. Why should linking libsystemd alone produce this issue if XZ is not otherwise used? AIUI, libsystemd is linked to get access to sd_notify, no xz required for that (nor any other SSH functionality). I'm guessing that liblzma (pulled in by libsystemd) has some initializer functions that get called even though its contents are not actually used. As I said, I haven't yet seen this bit explained.

 

[BarnSoftEng]

It should be noted that the attack only works because Debian and Redhat added functionality to sshd that is not present in it as distributed by its developers. The extra functionality adds systemd interaction, which requires libsystemd which requires liblzma, a component of the (compromised) xz package.

One should be wary of distributions adding functionality. Often it increases the attack surface, not only because of the modifications/additions themselves, but also by adding dependencies.

This conclusion ignores the huge value that the patches that all OS apply to packages provides. Without patches much upstream software will not build, will not run correctly.

 

[pseudonomous]

It should be noted that the attack only works because Debian and Redhat added functionality to sshd that is not present in it as distributed by its developers. The extra functionality adds systemd interaction, which requires libsystemd which requires liblzma, a component of the (compromised) xz package.

One should be wary of distributions adding functionality. Often it increases the attack surface, not only because of the modifications/additions themselves, but also by adding dependencies.

While this is true, to a degree, it's also quite possible that if there were a lot of interesting targets out there that didn't patch sshd to use systemd and libxz, that given the level of access they had, the actor here could almost certainly found some other way to put an obfuscated backdoor into the code.

I mean, certainly the OpenSSH project is obviously in the clear, but having gained commit access to a critical library, I don't think we could feel safe that "Jia Tan" couldn't have constructed a different backdoor even if they couldn't use the particular behavior Debian and Ubuntu added to their version of SSH.

And I'm not sure anybody feels totally safe until somebody audits the code to make sure there's nothing else lurking in here that might be dangerous.

 

[BarnSoftEng]

So a prime reason this became potentially exploitable is libsystemd in OpenSSH. Need I say more.

The prime reason is a very well funded and capable attacker looked for a way in.
if not xz or systemd then they would have attacked via the next candidate weak point.

 

[AnotherAnonymousCoward]

Shocking to see that this was only discovered by accident. This presumably means that the software we are using is full of backdoors ...

 

[om1]

This conclusion ignores the huge value that the patches that all OS apply to packages provides. Without patches much upstream software will not build, will not run correctly.

There's a difference between patches to make it build and adding stuff. Also you underestimate the quality of the ssh code, it is highly portable.

 

[siddhartht]

The whole world got lucky that one developer was determined enough to discover the cause of a minor performance regression. Just makes you wonder what else hasn't yet been discovered in our open source tooling out there.

Really eye opening, thank you Andres!

Edit: And of course, Dan for the wonderful article.

I should also point out that it is (hyper) common for scientific packages to reuse other packages. In the statistics and bioinformatics world, we use R a lot - let’s say if I’m going to install a common Bioinformatics package DESeq2, it has more than 8 dependencies - all those dependencies in turn may have more dependencies and the dependency tree gets more dense with each point being a potential source of malicious code.

Code and library reuse is not bad - but I have seen scientists (and I am also to blame here) use external libraries to do simple tasks like making plots which can be done in base R. The resulting workflow then is a mess of unnecessary dependencies because I/someone was too lazy to reuse built in functions and write perhaps a new function or read the developer reference. While I’m using R as an example here, Python, Julia and Rust(cargo) are equally susceptible to (malicious)code/library (re)use.

Most of the time, we use scientific software in a cluster - which depending on how well the system admins have configured the whole setup, may allow access to data from other researchers - and a compromised toolchain used to make the package I use can cause damage not only to my work, but to broader scientific community using that computational resource.

I do not know the solution - perhaps bundling all dependencies together - a la conda environment - but that would also bundle the bugs specific to the versions used, or perhaps going deeper and only reusing the code which does that task I need, which will require a lot more time and expertise. Or maybe I should just trust the package maintainers…

 

[om1]

There's a bit about this I haven't seen explained. Why should linking libsystemd alone produce this issue if XZ is not otherwise used? AIUI, libsystemd is linked to get access to sd_notify, no xz required for that (nor any other SSH functionality). I'm guessing that liblzma (pulled in by libsystemd) has some initializer functions that get called even though its contents are not actually used. As I said, I haven't yet seen this bit explained.

When a lib is loaded, it's init code is executed. Te patched liblzma then uses ifunc to patch the running sshd

 

[om1]

The prime reason is a very well funded and capable attacker looked for a way in.
if not xz or systemd then they would have attacked via the next candidate weak point.

I do know that the OpenSSH devs are very strict on not increasing attack surface. The main reason why they did not want to link to systemd in the distributed version. You can be sure that OpenSSH as distributed contains less opportunity for attackers than the modified versions. That's my main point: modifications by distributions are not always improvements.

 

[serafean]

It seems we still know very little about the actual payload itself, or is Ars deliberately sparse on details?
I haven't yet seen a detailed analysis of what the hijacked functions actually do. Accepting a master key is kind of a given, but the fact it also has a kill switch [1] is interesting in and of itself...

[1] https://piaille.fr/@zeno/112185928685603910

 

[zogus]

The key point begins the last paragraph and is comprehensible to all.

"What more do we know about Jia Tan?
At the moment, extremely little, especially for someone entrusted to steward a piece of software as ubiquitous and as sensitive as xz Utils."

Trust mysterious actors? Elementary folly.

It’s a lot more complicated than that.

The FOSS ideal is “You are what you code,” not “Your reputation precedes you.” As such, it shouldn’t matter one whit if you are a seventeen-year cicada larva living beneath a tree in Aurora, IL, if your code passes upstream review. But in reality, the “review” often ends up becoming proof by prior achievement, because of time and complexity, and we get into situations where bad actors abuse their trust. It’s exactly the same as the situation surrounding science peer review.

It’s easy enough to point at the affair and scream: “Aha! So we need less coders and more testers!” The problem is, no one wants to be the tester. It’s exactly like the problem surrounding peer-reviewed science papers: the flow of fraudulent papers never stops because no one qualified to do peer reviews wants to spend all their time doing it.

 

[leonwid]

The key point begins the last paragraph and is comprehensible to all.

"What more do we know about Jia Tan?
At the moment, extremely little, especially for someone entrusted to steward a piece of software as ubiquitous and as sensitive as xz Utils."

Trust mysterious actors? Elementary folly.

And almost the definition of an open source contributor.

I have not seen many open source projects that require and ID and address verification step before committing. And while something will need to happen to guard a bit better against supply chain attacks I don’t think the fix will be extensive vetting.

 

[gautier]

"This developer persona has touched dozens of other pieces of open-source software in the past few years.". Well, I guess the Opensource community have some codes to review. Maybe the xz incident is only the tips of the iceberg.

 

[bbf]

Open source software is only as good as the people that review the check-ins.

I guess a good first step would be a more robust system for updating the "official" maintainer of each open source software project that's included in repositories installed by default in these big linux distros.

But definitely it's going to be hard to completely button up everything due to the large amount of software used in a distro. The well equipped black hat can and will find the weakest package to exploit.

 

[MikeGale]

We are lucky that this was detected and that some competent people have moved in to analyse. I presume more analysis is still being done. (Thanks to them.)

What we don't know is how many other similar attacks have been deployed, and how many are in preparation.

What can decent people do to reduce our risks?

Thoughts that come to mind:
1. Some of those who do this, often thankless, maintenance work might like more support. This might be financial contributions, or people (who know what they're doing) reviewing code submissions. Those who incorporate these libraries into their own programs (and all users) should maybe think about this. If there were a "donate to the maintainers" button on a story like this, that would convert the immediate story into something of greater value, if the maintainer would like that.
2. Some of the maintainers might appreciate recognition. Some won't, but worth considering.
3. Some who use the libraries can improve the checking they do.
4. Unpleasant people who harass maintainers should be detected and treated appropriately.

 

[MrPractical]

A small error to fix: Jia Tan's username is referred to twice as JiaT575 and twice as JiaT75.

 

[TheMongoose]

"This developer persona has touched dozens of other pieces of open-source software in the past few years.". Well, I guess the Opensource community have some codes to review. Maybe the xz incident is only the tips of the iceberg.

Right? I didn’t realise until I read this article just how far back this started. Now you’ve got to wonder: if this was a group, how many other so-far benign identities they have, contributing good code and building up reputations. And how easy it would be for them to slip a little something into another fundamental tool.

 

[LosD]

So a prime reason this became potentially exploitable is libsystemd in OpenSSH. Need I say more.

No need, you proved all we needed to know about you with that sentence alone.

 

[trie]

For instance, Tan replaced Collins's contact information with their own on Microsoft's oss-fuzz, a project that scans open-source software for signs of maliciousness.

1. Fuzzing scans for programming errors/vulnerabilities, which can be exploited maliciously, not "signs of maliciousness"
2. Afaik, Microsoft does not have a fork of oss-fuzz

 

[O_mia_bela_Madonina]

It’s easy enough to point at the affair and scream: “Aha! So we need less coders and more testers!” The problem is, no one wants to be the tester. It’s exactly like the problem surrounding peer-reviewed science papers: the flow of fraudulent papers never stops because no one qualified to do peer reviews wants to spend all their time doing it.

I agree, but this does not appear to be an unsolvable problem. There are lots of people who spend their entire working life "simply" testing things like food, fire equipment, working conditions, etc. Apparently we have yet to accept the fact that software and scientific papers are critical tools today.

 

[Riot Nrrrdâ„¢]

On Friday, researchers revealed the discovery of a backdoor [...]

No, one single Postgres developer (not a "researcher") - Andres Freund - revealed the discovery of a backdoor. Everybody else piled in to help, but he is the sole 'discoverer'. Credit where credit is due. (And everyone should buy Andres a beer.)

 

[barrattm]

Shocking to see that this was only discovered by accident. This presumably means that the software we are using is full of backdoors ...

It strikes me that downvoters of your post are probably delusional. There's much about the world of OSS software that pivots on people being nice, and there's many that hold dearly to the notion that everyone is. We've just witnessed the (near) consequence of blind trust in some well established project.

There are literally thousands of individual developers of across the myriad security-critical packages that are important to the average Linux distro. It's inconceivable that they're all invulnerable to malign influence.

What we've just seen is a maintainer of a well-established and well respected package get hoodwinked by some pretty basic social engineering preying on the fact that they're not so active with their work as they once were. Everyone gets somewhat tired of maintaining their package, especially when it's been "complete" and "trouble free" for a long time already.

There must be a lot of packages in that state. Control of many such "mission critical" packages is going to get handed on to some successor, and it's kind of important that the successor is trustable. So perhaps we should brace for more such events happening (if they haven't already occured).

Other Bad Outcomes
With "safe succession" of package control becoming a vital discussion point, what we might see is an organisation (lets call it "RedHat") offering to take over control. It'd be a "Trust us, we're RedHat, pure as the driven snow" offer to independent package controllers looking to retire.

Of course, what that means is that more and more of the OSS packages that everyone depends on comes under the control of an organisation that already has demonstrated a willingness to leverage such control for its own profit.

If we don't want such an outcome, someone else is probably going to have adopt packages. The problem is that such an effort would require volunteers, trustworthy volunteers. What we've just seen is a "volunteer" who wasn't trustworthy get the keys.

 

[mustached-dog]

Interestingly enough, "Jia Tan" is very close to 加蛋 in Mandarin, meaning "to add an egg". Unlikely to be a real name or a coincidence.

 

[Excors]

There's a bit about this I haven't seen explained. Why should linking libsystemd alone produce this issue if XZ is not otherwise used? AIUI, libsystemd is linked to get access to sd_notify, no xz required for that (nor any other SSH functionality). I'm guessing that liblzma (pulled in by libsystemd) has some initializer functions that get called even though its contents are not actually used. As I said, I haven't yet seen this bit explained.

It uses glibc's ifunc mechanism. That lets a library provide multiple implementations of a function (typically optimised for different CPUs), plus a resolver function which chooses one of those implementations. When the library is loaded, the loader will execute the resolver, and then any calls to the ifunc will efficiently jump to the chosen implementation.

XZ was patched last year to make lzma_crc64 an ifunc. (It has an optimised implementation for CPUs with the CLMUL instruction, so it seems quite reasonable to select that at load-time and skip an indirect branch on every function call.)

The backdoored build script subtly modifies crc64_fast.c so the resolver calls a function named _get_cpuid(), instead of the usual __get_cpuid() (with two underscores). _get_cpuid is implemented in a binary .o file (hidden in test data) and initialises the runtime backdoor, intercepting calls to RSA_public_decrypt.

There are much easier ways to execute code at load-time (you can simply declare a function with __attribute__((constructor)) and insert that via the backdoored build script) - it looks like the approach with ifuncs was chosen for obfuscation. Even if someone noticed liblzma executing code at load-time in sshd and got suspicious, they'd trace it and see that it was crc64_resolve calling _get_cpuid, and they'd look at the git log and see an entirely plausible reason for that code, and probably they'd accept that and wouldn't investigate any further.

Incidentally, systemd has (for unrelated reasons) recently changed to load libraries on demand with dlopen(), so liblzma would not be loaded until the application called an API that required it (in this case the API for reading compressed logs, which sshd will never do). That would have stopped this specific attack, though likely the attacker would have just switched to a different attack - no technical measures can protect you from the trusted maintainer of such a widely-used library.

 

[Flying Officer Ives]

4. Unpleasant people who harass maintainers should be detected and treated appropriately.

So like the guy who said the package wasn’t being updated fast enough or often enough? It’s sad that a few new accounts or non-contributors can so easily put pressure on a maintainer.

 

[zogus]

I agree, but this does not appear to be an unsolvable problem. There are lots of people who spend their entire working life "simply" testing things like food, fire equipment, working conditions, etc. Apparently we have yet to accept the fact that software and scientific papers are critical tools today.

"Working" is the important word here. As far as I can tell from reading the news, a big reason this exploit almost succeeded was because no one was getting paid to work on xz. "So pay 'em!" you say. Where is the money gonna come from?

 

[malor]

It seems we still know very little about the actual payload itself, or is Ars deliberately sparse on details?
I haven't yet seen a detailed analysis of what the hijacked functions actually do. Accepting a master key is kind of a given, but the fact it also has a kill switch [1] is interesting in and of itself...

[1] https://piaille.fr/@zeno/112185928685603910

As far as I can tell from the writeup, the payload is that a person with the master key can get a root shell on any distro running SSH on a port they can reach, while having the bogus version of xz installed.

That's it, that's the payload, instant remote root access, which could have extended to millions of computers. Every machine broken into could have been exploited in any way the master key user wanted. That user would probably have had a standard toolkit to deploy, but that's just follow-on exploitation.

 

[choco bo]

Performance hit is quite substantial, actually. I have no doubt that this thing would have been detected, eventually. However, it might have happened months from now. Then it would have been everywhere already.

But this is a good thing. A very good thing, actually.

There have been discussions about supply chain attacks, for years. Decades, actually. We used to call it "poisoning the well" many years ago. But no matter how much we talk about it, it was all theoretical. I mean, people even assumed that compilers have been backdoored many years ago. But noone was going to spend this much effort just to show that it was possible and to make people accept the possibility. So not much was really done about it.

Until now.

Now we are already seeing changes being made to OpenSSH that would have not been possible few months ago. Native systemd notification integration is already been developed (since 30th of March), so no need for libsystemd linking anymore. It will take some time to get integrated but it will happen. We are seeing people understanding that there is absolutely no need to have binary blobs in source repositories (except rare cases, of course, but those are going to be audited even more now). Checking source repositories against tarballs have been done before, many times. But obviously it wasn't good enough or often enough. That will change as well. People being dicks to maintainers are going to get greeted with "go fuck yourself" now, without a second thought. It will be extreme but it will be safer. For eternity I was terrified of compiling software myself because every time I invoked "./configure ..." I would think "fuck knows what is going on there right now". I did occasionally check scripts, I would grep for unexpected things but I was aware I'd never detect a very skilled attacker, like this one. Now there is going to be much more checking of autoconf/make/CMake/etc files in source repos. It won't be easy to detect things, but it will be easier. More eyes will be put on sources. For example, I am going to pick a random smaller project and just read the commit history, look for oddities, etc. Not because I expect to find something but I want to see what else should be looked at, etc. Eventually, I might end up with toolset that might help speed this process up. So there will be at least one more set of eyes looking at sources. I imagine that companies/organizations with more resources are going to put tons of effort into automating all this. So yeah, xz backdoor is actually a good thing, in a very bizarre way.

Also, I can't hunt all the references at the moment but I believe it was certificate (not the SSH key) that is used as a vector of attack, because certs are checked early and no configuration options will disable that check, while it wouldn't be the case with keys. A change to OpenSSH has already been suggested so OpenSSH will only get more secure because of this and one less vector of attack is now available.

Amount of skill and time/effort invested in this is mind blowing. I don't think people outside security really comprehend the skill/time involved here, this was insanely well executed attack. My first thought was "This had to be TURLA" because it was insanely smart and whoever did this had lots of patience. This does not (and will not) happen often.

So yeah, we were incredibly lucky that a Postgres developer caught it early.

However, it is mind blowing how many times security incidents have been detected by looking at CPU/RAM usage on systems, it is really no surprise that this is how xz backdoor got detected.

 

[uberDoward]

When a lib is loaded, it's init code is executed. Te patched liblzma then uses ifunc to patch the running sshd

Wouldn't this mean any software loading the vulnerable version(s) of liblzma would have targeted a running sshd instance?