The Man Who Saved The World

The Man Who Saved The World

We see in the news all the time; hacker this, hacker that and then we see something drastic happen and wonder why more people didn’t pay attention to prevent it from happening again.

Case in point would be the Solarwinds supply chain attack that wrecked havoc for 10’s of thousands of customers, yet, the media squawked like fools, the tech industry placed blame wherever they could and in the end, exactly like how politicians deal with problems, make a new rule, tell the public that they did something and then sweep it under the rug, never to be heard from again.

In this case, yet another supply chain attack involving the XZ Utils. XZ Utils, and its underlying library liblzma, are open-source projects that implement the lzma compression and decompression. They are included in many Linux distributions out of the box and are very popular with developers, and are used extensively throughout the Linux ecosystem.

Almost two years ago, a developer under the name of Jia Tan joined the project and started opening pull requests for various bug fixes or improvements. So far, nothing is out of the ordinary; this is how things work in the open-source world. Eventually, after building trust and credibility, Jia Tan began to receive permissions for the repository — first, commit permissions and, eventually, release manager rights.

It seems that as part of the effort to gain these permissions, Jia Tan used an interesting form of social engineering. They used fake accounts to send a myriad of feature requests and complaints about bugs to pressure the original maintainer, Lasse Collin, (who at the time was dealing with a demanding job) eventually causing the need to add another maintainer to the repository…

After contributing to the code for approximately two years, in 2023 Jia Tan introduced a few changes to XZ that were included as part of release 5.6.0. Among these changes was a sophisticated backdoor.

The backdoor is quite complex. For starters, you won’t find it in the xz GitHub repository (which is currently disabled, but that’s besides the point). In what seems like an attempt to avoid detection, instead of pushing parts of the backdoor to the public git repository, the malicious maintainer only included it in source code tarball releases. This caused parts of the backdoor to remain relatively hidden, while still being used during the build process of dependent projects.

The backdoor is composed of many parts introduced over multiple commits. Using IFUNCs in the build process, which will be used to hijack the symbol resolve functions by the malware. Including an obfuscated shared object hidden in test files. Running a script set during the build process of the library that extracts the shared object (not included in the repository, only in releases, but added to .gitignore). Disabling landlocking, which is a security feature to restrict process privileges.

The execution chain also consists of multiple stages. The malicious script build-to-host.m4 is run during the library’s build process and decodes the “test” file bad-3-corrupt_lzma2.xz into a bash script. The bash script then performs a more complicated decode process on another “test” file, good-large_compressed.lzma, decoding it into another script. That script then extracts a shared object liblzma_la-crc64-fast.o, which is added to the compilation process of liblzma.

This process is admittedly hard to follow. We recommend Thomas Roccia’s infographic for a great visual reference and in-depth analysis.

The shared object itself is compiled into liblzma, and replaces the regular function name resolution process. During any process loading, function names are resolved into actual pointers to the process memory, pointing at the binary code. The malicious library interferes with the function resolving process, so it could replace the function pointer for the OpenSSH function RSA_public_decrypt.

It then points that function to a malicious one of its own, which according to research published by Filippo Valsorda, extracts a command from the authenticating client’s certificate (after verifying that it is the threat actor) and passes it on to the system() function for execution, thereby achieving RCE prior to authentication. The malicious library interferes with the function resolving process, so it could replace the function pointer for the OpenSSH function RSA_public_decrypt.

For a more detailed explanation of the backdoor parts, you can read Andres Freund’s post on openwall.

Currently, it appears as though the backdoor is added to the SSH daemon on the vulnerable machine, enabling a remote attacker to execute arbitrary code. This means that any machine with the vulnerable package that exposes SSH to the internet is potentially vulnerable.

This backdoor almost became one of the most significant intrusion enablers ever — one that would’ve dwarfed the SolarWinds backdoor. The attackers were almost able to gain immediate access to any Linux machine running an infected distro, which includes Fedora, Ubuntu, and Debian. Almost.

There was only one thing that stopped that from happening — Andres Freund. After investigating a 500 ms latency issue that was introduced after a software update, Andres was able to trace the issue back to the xz package and ultimately identify the backdoor.

This obviously raises a lot of concerns. We got lucky. If this backdoor was not detected by a curious engineer, how long would it have remained active?

And perhaps even more concerning. What if this has happened before?

The Cybersecurity and Infrastructure Security Agency (CISA) recommended course of action is to downgrade to an un-compromised version, such as 5.4.6.

To know which version of XZ Utils or liblzma you currently have on your systems, you can run the following query in by opening a terminal window and typing “xz –version.”  If the results are version 5.6.0 or 5.6.1, then you should roll-back to a lessor, more stable version immediately or if needed, remove those packages completely, hunt for any rouge files left over, delete those and then do a clean install of the XZ-Utils package.

Since the backdoor actually executes system commands, and isn’t just allowing authentication, it might be possible to detect this behavior via process tracking.

Usually, during logon, a new shell is created for the logging user, and runs the default shell process. However, with this backdoor, the malicious command is actually executed by the SSH daemon process, sshd, which could trigger an anomaly.

According to some analyses of the backdoor, it appears to have an environment variable kill switch. Adding the key yolAbejyiejuvnup=Evjtgvsh5okmkAvj to the system’s environment variables may disable the backdoor.

Now that we have all that out of the way, which Linux distributions were effected by this fiasco.

While serious, the impact may be limited. The problematic code is in the newer versions of xz/liblzma, so it may not be as widely deployed. Linux distributions that have not yet released the newer versions are less likely to be affected.

Red Hat: Vulnerable packages are present in Fedora 41 and Fedora Rawhide. No versions of Red Hat Enterprise Linux (RHEL) are affected. Red Hat says users should immediately stop using the affected versions until the company has had a chance to change the xz version.

SUSE: An update is available for openSUSE (Tumbleweed or MicroOS).

Debian Linux: No stable versions of the distribution are affected, but compromised packages were part of the testing, unstable, and experimental versions. Users should update xz-utils.

Kali Linux: If systems were updated between March 26 and March 29, then users should update again to get the fix. If Kali’s last update was before March 26, it is not affected by this backdoor.

Ubuntu: One of the most popular Linux distributions is unaffected as its version of xz is the older 5.4 version. No update is required.

In the end, we can thank Andres Freund for saving the world, a developer and engineer working on Microsoft’s PostgreSQL offerings, while troubleshooting performance problems on a Debian system that he was experiencing with SSH, the most widely used protocol for remotely logging in to devices over the Internet. Specifically, SSH logins were consuming too many CPU cycles and were generating errors with valgrind, a utility for monitoring computer memory.