The Other Players Who Helped (Almost) Make the World’s Biggest Backdoor Hack

On March 29, Microsoft software developer Andres Freund was trying to optimize the performance of his computer when he noticed that one program was using an unexpected amount of processing power. Freund dove in to troubleshoot and “got suspicious.”

Eventually, Freund found the source of the problem, which he subsequently posted to a security mailing list: He had discovered a backdoor in XZ Utils, a data compression utility used by a wide array of various Linux-based computer applications — a constellation of open-source software that, while often not consumer-facing, undergirds key computing and internet functions like secure communications between machines.

By inadvertently spotting the backdoor, which was buried deep in the code in binary test files, Freund averted a large-scale security catastrophe. Any machine running an operating system that included the backdoored utility and met the specifications laid out in the malicious code would have been vulnerable to compromise, allowing an attacker to potentially take control of the system.

The XZ backdoor was introduced by way of what is known as a software supply chain attack, which the National Counterintelligence and Security Center defines as “deliberate acts directed against the supply chains of software products themselves.” The attacks often employ complex ways of changing the source code of the programs, such as gaining unauthorized access to a developer’s system or through a malicious insider with legitimate access.

The malicious code in XZ Utils was introduced by a user calling themself Jia Tan, employing the handle JiaT75, according to Ars Technica and Wired. Tan had been a contributor to the XZ project since at least late 2021 and built trust with the community of developers working on it. Eventually, though the exact timeline is unclear, Tan ascended to being co-maintainer of the project, alongside the founder, Lasse Collin, allowing Tan to add code without needing the contributions to be approved. (Neither Tan nor Collin responded to requests for comment.)

The XZ backdoor betrays a sophisticated, meticulous operation. First, whoever led the attack identified a piece of software that would be embedded in a vast array of Linux operating systems. The development of this widely used technical utility was understaffed, with a single, core maintainer, Collin, who later conceded he was unable to maintain XZ, providing the opportunity for another developer to step in. Then, after cultivating Collin’s trust over a period of years, Tan injected a backdoor into the utility. All these moves were underlaid by a technical proficiency that ushered the creation and embedding of the actual backdoor code — a code sophisticated enough that analysis of its precise functionality and capability is still ongoing.

“The care taken to hide the exploits in binary test files as well as the sheer time taken to gain a reputation in the open-source project to later exploit it are abnormally sophisticated,” said Molly, a system administrator at Electronic Frontier Foundation who goes by a mononym. “However, there isn’t any indication yet whether this was state sponsored, a hacking group, a rogue developer, or any combination of the above.”

Tan’s elevation to being a co-maintainer mostly played out on an email group where code developers — in the open-source, collaborative spirit of the Linux family of operating systems — exchange ideas and strategize to build applications.

On one email list, Collin faced a raft of complaints. A group of users, relatively new to the project, had protested that Collin was falling behind and not making updates to the software quickly enough. He should, some of these users said, hand over control of the project; some explicitly called for the addition of another maintainer. Conceding that he could no longer devote enough attention to the project, Collin made Tan a co-maintainer.

The users involved in the complaints seemed to materialize from nowhere — posting their messages from what appear to be recently created Proton Mail accounts, then disappearing. Their entire online presence is related to these brief interactions on the mailing list dedicated to XZ; their only recorded interest is in quickly ushering along updates to the software.

Various U.S. intelligence agencies have recently expressed interest in addressing software supply chain attacks. The Cybersecurity and Infrastructure Security Agency jumped into action after Freund’s discovery, publishing an alert about the XZ backdoor on March 29, the same day Freund publicly posted about it.

Open-Source Players

In the open-source world of Linux programming — and in the development of XZ Utils — collaboration is carried out through email groups and code repositories. Tan posted on the listserv, chatted to Collin, and contributed code changes on the code repository Github, which is owned by Microsoft. GitHub has since disabled access to the XZ repository and disabled Tan’s account. (In February, The Intercept and other digital news firms sued Microsoft and its partner OpenAI for using their journalism without permission or credit.)

Several other figures on the email list participated in efforts — appearing to be diffuse but coinciding in their aims and timing — to install the new co-maintainer, sometimes particularly pushing for Tan.

Later, on a listserv dedicated to Debian, one of the more popular of the Linux family of operating systems, another group of users advocated for the backdoored version of XZ Utils to be included in the operating system’s distribution.

These dedicated groups played discrete roles: In one case, complaining about the lack of progress on XZ Utils and pushing for speedier updates by installing a new co-maintainer; and, in the other case, pushing for updated versions to be quickly and widely distributed.

“I think the multiple green accounts seeming to coordinate on specific goals at key times fits the pattern of using networks of sock accounts for social engineering that we’ve seen all over social media,” said Molly, the EFF system administrator. “It’s very possible that the rogue dev, hacking group, or state sponsor employed this tactic as part of their plan to introduce the back door. Of course, it’s also possible these are just coincidences.”

The pattern seems to fit what’s known in intelligence parlance as “persona management,” the practice of creating and subsequently maintaining multiple fictitious identities. A leaked document from the defense contractor HBGary Federal outlines the meticulousness that may go into maintaining these fictive personas, including creating an elaborate online footprint — something which was decidedly missing from the accounts involved in the XZ timeline.

While these other users employed different emails, in some cases they used providers that give clues as to when their accounts were created. When they used Proton Mail accounts, for instance, the encryption keys associated with these accounts were created on the same day, or mere days before, the users’ first posts to the email group. (Users, however, can also generate new keys, meaning the email addresses may have been older than their current keys.)

One of the earliest of these users on the list used the name Jigar Kumar. Kumar appears on the XZ development mailing list in April 2022, complaining that some features of the tool are confusing. Tan promptly responded to the comment. (Kumar did not respond to a request for comment.)

Kumar repeatedly popped up with subsequent complaints, sometimes building off others’ discontent. After Dennis Ens appeared on the same mailing list, Ens also complained about the lack of response to one of his messages. Collin acknowledged things were piling up and mentioned Tan had been helping him off list; he might soon have “a bigger role with XZ Utils.” (Ens did not respond to a request for comment.)

After another complaint from Kumar calling for a new maintainer, Collin responded: “I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we’ll see.”

The pressure kept coming. “As I have hinted in earlier emails, Jia Tan may have a bigger role in the project in the future,” Collin responded after Ens suggested he hand off some responsibilities. “He has been helping a lot off-list and is practically a co-maintainer already. :-)”

Ens then went quiet for two years — reemerging around the time the bulk of the malicious backdoor code was installed in the XZ software. Ens kept urging ever quicker updates.

After Collin eventually made Tan a co-maintainer, there was a subsequent push to get XZ Utils — which by now had the backdoor — distributed widely. After first showing up on the XZ GitHub repository in June 2023, another figure calling themselves Hans Jansen went on this March to push for the new version of XZ to be included in Debian Linux. (Jansen did not respond to a request for comment.)

An employee at Red Hat, a software firm owned by IBM, which sponsors and helps maintain Fedora, another popular Linux operating system, described Tan trying to convince him to help add the compromised XZ Utils to Fedora.

These popular Linux operating systems account for millions of computer users — meaning that huge numbers of users would have been open to compromise if Freund, the developer, had not discovered the backdoor.

“While the possibility of socially engineering backdoors in critical software seems like an indictment of open-source projects, it’s not exclusive to open source and could happen anywhere,” said Molly. “In fact, the ability for the engineer to discover this backdoor before it was shipped was only possible due to the open nature of the project.”

The post The Other Players Who Helped (Almost) Make the World’s Biggest Backdoor Hack appeared first on The Intercept.