A sophisticated backdoor targeting the SSH service on Linux was made against the XZ compression library in a supply chain attack. The backdoor almost made it into most major Linux distributions until a sharp-eyed engineer saw a problem during testing and dug into the issue deeper.

The technical details of the attack have been covered in other articles, but in sum:

• Attackers used a 2+ year long timeline to build trust with the maintainer of the XZ library.

• The attackers managed to work over time to add in changes, finally culminating into an obfuscated and sophisticated backdoor inserted into the build of the library.

• The XZ tools package, when built, would output a shared library that was then used by the system SSH daemon (among others).

• The shared library contained code to override SSH authentication functions if presented with a correct cryptographic signature on a payload.

• Once the signature was validated, the SSH daemon would execute the attacker payload, bypassing all authentication, logging, and other security measures.

Who did the attack is still unknown, but the code had a very high level of sophistication and operational planning. By targeting SSH, the attackers would have had a master key for most major distributions of Linux. The attack was bold and well planned, with sheer luck causing it to be discovered in the end.

Affected Hosts

The attack affected users of bleeding edge Linux distributions using only the absolute newest libraries. Generally, these are what are termed "rolling releases" or unstable releases used mostly for testing and development. It is unlikely production systems would have been impacted. At this time, the following distributions would have been affected:

• Fedora (unstable)

• Debian (unstable)

• OpenSuse Tumbleweed (rolling release)

• Kali (a pentest distribution)

Affected, but not likely vulnerable due to how SSH is built:

• Arch (rolling release)

Most users would likely not have these systems in production or on Internet facing environments where SSH would have been subjected to attack.

Backdoor Operation

When an attacker connects to a vulnerable SSH server, they can send over a payload that has the correct cryptographic signature. If the signature matches, the payload is sent directly to the system() call on the Linux host to be run with the same rights as SSH (usually root superuser).

In the process above, there is no logging of the attack. The cryptographic signature validation is a critical component as well. It means that the attacker can have a very high assurance that only they would be able to execute the backdoor when needed. The capability could not be used by any other person without the correct signing key.

Backdoor Detection with Suspicious Process Environment

Sandfly shies away from doing file scanning for specific malware (which is unreliable on Linux due to the open source nature). Instead, we like to focus on malware tactics and techniques that are more difficult to evade. In this case, we focus on the results of spawned processes from the backdoor as they have certain traits that are unusual.

The main trait when a process is started by the XZ backdoor, is that it has an environment that is inherited from the SSH daemon itself instead of a more standard looking user or system process.

Below we show some Linux command line forensics investigating a spawned process from the XZ backdoor on Debian. The process environment shows elements that are clearly related to the SSH daemon and references to systemd. These are not what you'd expect to see from a process started from a normal SSH login shell (or even started as a direct command over an SSH connection).

The command used is:

strings /proc/<PID>/environ

Where <PID> is the process ID you wish to investigate.

Above we see a few oddities in the environment in an affected Debian host:

• SSHD_OPTS - Holds SSH daemon options in some configurations.

• RUNTIME_DIRECTORY - Points to directories that have SSH daemon runtime information.

Other systems like Fedora have different environment variables, but the same suspicious link to the SSH daemon:

• MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/sshd.service/memory.pressure

In the above the reference to the sshd.service would be unusual.

Here we see how this environment looks in a process seen by Sandfly:

Backdoor Detection with Sandfly

We have made some signatures to detect processes started directly from the SSH daemon that are backdoored. It can potentially spot other backdoors operating with similar tactics. Below are some examples where we executed a sleep and netcat process via the XZ backdoor to show how detection would look.

XZ Backdoor Sandfly Custom Signatures

We have made a repository for fully-licensed customers wishing to try out these signatures today. We will be making this detection part of our standard security modules on our next release for all users:

Sandfly Security Preview Repository

XZ Backdoor and Agentless Drift Detection

Sandfly's new agentless drift detection can spot processes and other changes made to a system run through the XZ backdoor (or other backdoors). In the case of the XZ backdoor in particular, processes started by the malware were immediately detected by our process drift detection. If you have not tried Sandfly 5.0 and this new feature, we encourage you to check it out.

Backdoors are Useless without Pressing the Attack

While this backdoor is severe, it should be noted that there are a myriad of ways to backdoor a Linux system. It can happen in a very sophisticated way as we saw with the XZ backdoor, or very simply by compromising a web server and uploading a malicious script. In fact, Linux itself has many commands that can be used for backdoor purposes built into the OS if someone desires.

A backdoor is useless by itself. It must be used to press an attack in some way or another. What do we mean by this? Simply what will the attacker actually do with the backdoor? Having it sit there is not helping. Will they:

• Insert a new user?

• Tamper with log files?

• Start a malicious process for lateral movement?

• Exfiltrate data?

• Launch credential theft attacks?

They are going to do something and this is where defenders stand the best chance of detecting a breach.

Intruders use backdoors to further their objectives. As defenders this is where you will find they are the most exposed. This is why detecting the post-compromise activity is extremely important.

We encourage customers therefore to not get too distracted on the XZ backdoor itself. While it was a serious and interesting attack, it's not the only backdoor ever put into a software package and won't be the last. Keeping a vigilant eye on your systems for threat activity means that even with new backdoors and other attacks, you have the best chance of rapidly discovering compromised Linux systems before the damage becomes severe.