The XZ Utils Backdoor: Potentially Catastrophic Cyber Security Incident Averted

xz utils is a widely-used open-source data compression software that comes pre-installed on most Linux systems. It’s commonly used for compressing software packages, archives, kernel images, and more. In March 2024, Andres Freund, a Microsoft engineer, discovered that recent versions of xz utils contained an intentional backdoor. This malicious code could allow an attacker to remotely execute commands on affected systems, bypassing authentication.

The xz utils backdoor is particularly concerning due to its widespread impact, as xz utils is installed on almost every Linux distribution by default. The backdoor was also very sophisticated, expertly hidden in the build process using novel techniques. It allows remote code execution as root, potentially giving attackers total control of compromised systems. Evidence suggests the attackers worked on this backdoor since 2021, and it narrowly missed inclusion in several major distribution releases.

Timeline and Key Events

The xz utils backdoor was developed over several years by a sophisticated attacker:

In 2021, an account named “JiaT75” was created on GitHub and made some initial code contributions to open-source projects, including one to libarchive that was later deemed suspicious.

Throughout 2022, JiaT75 started participating in the xz utils mailing list, submitting patches, and pressuring the project to add a new maintainer. Lasse Collin, the original xz utils maintainer, appears to have been deliberately targeted and manipulated.

In 2023, JiaT75, under the name “Jia Tan”, became a maintainer of xz utils, gained commit access, and made subtle changes laying the groundwork for the backdoor.

In early 2024, Tan made changes to prevent detection of the backdoor code and merged the backdoor disguised as new “test files”. Sock puppet accounts immediately started pushing for the backdoored version to be included in various Linux distributions.

On March 29, 2024, Andres Freund discovered and reported the backdoor. The security community quickly responded to analyze the backdoor and protect systems. Tan was removed from the xz utils project and the malicious code was reverted.

How the Backdoor Works

The xz utils backdoor is a sophisticated multi-stage exploit that hijacks the build process to inject malicious code. It takes advantage of xz’s “make” process by using a modified configuration script that reads a malicious “test file” containing obfuscated shell commands. These commands are executed during the build, resulting in a compromised xz binary.

The backdoor code is hidden in two files disguised as test inputs for xz’s compression algorithm. The commands in these files are obfuscated using multiple layers of compression, encoding, and encryption. The backdoor includes checks to only trigger on 64-bit Linux systems with glibc, helping avoid detection.

A key component is the “liblzma_la-crc64-fast.o” object file, which contains code that hooks the “IFUNC” resolver, a glibc feature allowing libraries to redirect function calls. The backdoor uses this to intercept OpenSSH’s “RSA_public_decrypt” function, checks for a hardcoded “magic” value in the client’s RSA key, and allows authentication bypass if present.

The backdoor can also execute arbitrary commands using an encrypted payload in the malicious RSA key. If the payload decrypts correctly, the backdoor executes it with full root privileges.

Potential Impact

The xz utils backdoor had the potential to be one of the most damaging Linux security incidents due to its ability to completely compromise affected systems. With xz utils installed by default on most Linux distributions, hundreds of millions of machines across servers, cloud services, IoT devices, workstations, and appliances were at risk.

The backdoor was caught shortly before inclusion in stable releases of Fedora, Debian, and Ubuntu, which would have drastically increased the scale of the incident. Analysis of the payload is ongoing, and it’s possible that there are still undiscovered capabilities.

Response and Lessons Learned

The response to the xz utils backdoor was rapid, with the security community mobilizing quickly to analyze the backdoor, develop mitigations, and distribute patched versions. However, the incident highlights the need for improvements in the open source software ecosystem.

The attack demonstrates the significant impact a compromise in a critical open source component can have and the importance of improving the security of key projects and infrastructure. Open source projects should strengthen their processes for code review, maintainer vetting, and anomaly detection.

The multi-year timeline and sophisticated techniques used suggest the attacker was highly capable and persistent, potentially nation-state backed. Defending against such threats requires sharing threat intelligence and investing in monitoring and response capabilities.

Ultimately, this incident underscores the critical importance of open source security. Increased funding, research, collaboration, and treating open source security as a collective responsibility is essential for meaningful progress.

Conclusion

The xz utils backdoor is one of the most significant and sophisticated attacks on open source software to date. While the worst-case scenario was prevented due to its timely discovery and the rapid response from the security community, it serves as a stark reminder of the risks in the open source supply chain.

The incident should serve as a wake-up call for the open source community and its users to prioritize and invest in the security of this critical ecosystem. Improving code auditing, contributor vetting, anomaly detection, and overall resilience is crucial. Open source software is a vital public good that requires active, collective protection to defend against the ever-evolving threat landscape.

Sources: 

  1. Greenberg, A. (2024). “What We Know About the XZ Utils Backdoor That Almost Infected the World.” Ars Technica. Available at: https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/
  2. “XZ Utils Backdoor Incident.” Tukaani Project. (2024). Available at: https://tukaani.org/xz-backdoor/
  3. Sam, T. (2024). “Detailed Analysis of the XZ Utils Backdoor Code.” Gist GitHub. Available at: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
  4. “Technical Analysis of the XZ Utils Backdoor CVE-2024-3094.” Pentest Tools Blog. (2024). Available at: https://pentest-tools.com/blog/xz-utils-backdoor-cve-2024-3094

Leave a Reply

Your email address will not be published. Required fields are marked *