Sandfly Blog

Sandfly 5.5 announces AI analysis for security events, enhanced coverage for BPFDoor, and scanning performance upgrades.

We’ve implemented AI Large Language Model (LLM) integrations for all major vendors and also on-prem LLM solutions for customers. This is a very powerful new feature and is not just hype. We recommend customers read more below to see if it is a fit for your organization.

Further, we have expanded detection coverage for updated BPFDoor malware. This malware is still impacting many critical infrastructure providers and our new detections expand our already existing coverage. 

Finally, we’ve made performance improvements to areas such as scheduled scans that now feature a default “trickle” mode. This mode will spread out scans over the time window to lower network and host loads. 

AI for Linux Security

We often get asked by customers what a particular alert means, how to investigate it, etc. This is especially true for teams that have limited access to personnel with deep Linux forensics understanding. After watching and experimenting with the new generation of AI LLMs, we feel we're at a place where we can offer a powerful new AI security analyst feature for our customers that helps answer these questions.

Good Data for Good Results

At Sandfly we saw that many LLMs were getting a bad reputation because they were being asked to analyze and draw conclusions from bad data. LLMs with bad data will always provide bad results when asked to interpret it (the same way a human would). However, Sandfly is different for two very important reasons:

1. We are in total control of our data quality because we generate it ourselves.
2. We know what questions to ask of that data.

These points are critical for good and reliable LLM usage in security contexts.

Understand that the data we obtain for Linux forensics is sourced by us directly and is the best in the industry. Our exclusive focus on Linux means we are not distracted with other platforms or need to worry about what a third party is throwing over the wall for us to handle in random logs. Our tools are purpose-built to collect Linux forensics data accurately and fast without relying on anyone else. Issues around LLM hallucinations and incorrect analysis are minimized as a result.

AI Analysis in Action

Our AI analysis is available in the host and results sections. For host analysis the LLM will summarize the state of the system and incorporate any alerts into the report. For results, the AI can be passed one or more entries and it will analyze each and attach the report to the result data. Let's demonstrate on actual malware.

Analyzing BPFDoor with AI

We'll show how our new LLM integration works with our old friend BPFDoor. On our test system we have started BPFDoor Version 2. It is memory resident and hiding waiting for activation of the backdoor with a magic packet. Sandfly has found this suspicious process and it's part of the host results showing multiple alerts as seen below.

Initiating AI Assistance

To begin, we go to our host and activate the Analyze Host button. This button will only be active if you have put in an API key to your preferred LLM provider. This can be a commercial cloud offering like Gemini, OpenAI, or Grok. Or, it can be an internal version of an LLM you choose.

Our analysis is now returned. Sandfly has created a host summary along with relevant alert data in an easy to understand summary seen below. You'll see several sections that we'll discuss.

Brief and Overview

All analyses begin with a brief and overview section. The brief gives you a one sentence synopsis of the problem with a full summary of the situation in the overview. Below we see our host situation is neatly summarized and we have a suspicious process that was identified.

Details

The details section lays out specifics about what is happening in easy to interpret points. Our host has many significant problems identified and summarized by the LLM in a way operators can easily understand.

Investigation

Next, we provide several areas to help guide the operator on how to investigate the suspicious activity on the host. These suggestions are non-destructive and can help gather additional information to add to an incident report before deciding actions. The investigative commands suggested here will often be installed by default so there is no need to load additional tools on systems to do an investigation.

Recommended Actions

We will then provide a list of simple and again non-destructive actions operators may want to take. They can vary depending on the attack type, but often involve ideas around host isolation, gathering forensic data, and more.

Suggested Items to Investigate

Finally, we'll give you a list of other items we suggest you investigate that may not be directly part of the existing forensic data, but often provide valuable information to help investigate root cause analysis. These recommendations again are non-destructive and built-in commands on most Linux hosts or may be available in your security stack elsewhere (e.g. network activity to the host).

Result Analysis

Analyzing individual results follows a similar procedure to host analysis, but can provide more specific insights into individual alerts. Users can send in multiple alerts for parallel analysis. The analysis data will remain with the alert result until deleted.

Ask The AI Security Analyst

We allow operators to ask the LLM questions about alerts and leverage it as a virtual security analyst. The capabilities here are really opened up with our data and the results and advice are very good. Often the questions can bridge novice users up to expert-level questions to dive really deeply into compromise and malware activity on Linux.

Below we see an example of how asking further questions allows the LLM to refine and provide more details on its conclusion about our BPFDoor attack.

In-Depth Linux Forensics

Operators can ask for more details on any part of an alert and get detailed responses that can help understand a threat. For example, below we ask the LLM to tell us why the detected file descriptors are important. This kind of questioning can help security teams learn what to look for and why it matters.

Cloud and On-Prem LLM Options

Our testing with major vendor LLMs from OpeanAI, Google and xAI (Grok) is that they all deliver very consistent and useful results with data from Sandfly. However, some customers may not feel comfortable sending data to an external LLM, so we have the option to use on-prem versions that are compatible with the OpenAI API as well.

Customers can easily choose what provider they want to use in the server configuration screen like below.

AI LLM Use is Optional

Of course, customers still maintain the option of not using an LLM at all. This feature must be enabled with your own API key. Sandfly does not enable this feature by default, and as usual we never collect or analyze any customer data if you decide to use this feature. However, customers will be exposing their data to a potential third party if they enable this feature. So, please make sure organization policy allows this. Again, we fully support on-prem OpenAI API compatible solutions if you are already using them internally. Private instances of of Gemini, OpenAI and others are also supported again with API key access.

BPFDoor Updates

BPFDoor continues to stalk critical infrastructure providers and we continue to hear about it from customers that contact us. Although we originally wrote about this threat back in 2022, it has evolved to become more evasive and secure.

BPFDoor Sniffer Evasion

As detailed in Haxrob’s blog above, the new BPFDoor variants have evolved. The basic changes are:

1. Much larger as a result of using improved, but heavier, public/private key infrastructure. This change means the backdoor is not easily accessed by anyone but the person that installed it. It also means monitoring the encrypted communications is more difficult.
2. The new version made a subtle change to how it uses network sockets to sniff traffic. Instead of traditional SOCK_RAW, it uses a poorly documented SOCK_DGRAM feature that allows it to grab network traffic but avoid detection with conventional checks for raw packet socket sniffing.

We updated Sandfly so that packet socket types decode the protocol and protocol type being accessed to better identify these changes.

For instance, a packet socket type will show whether it is not just using SOCK_DGRAM or SOCK_PACKET types, but also if it is grabbing IPv4, IPv6 or all IP traffic. This enables Sandfly to not only flag sniffing traffic before, but what kind of traffic is being sniffed and how. This detection can help find other types of sniffing activity in a more refined way vs. before.

Below we see how these file descriptors are shown for BPFDoor Version 1 and Version 2. Version 1 on top uses the traditional SOCK_RAW type. Version 2 below uses SOCK_DGRAM as shown in the next image.

BPFDoor Detections

The new socket decoding has allowed us to add the following new checks for potentially malicious sniffing activity associated with BPFDoor and others:

process_running_sniffer_operating_ipv4_traffic - Looks for processes sniffing all IPv4 traffic.

process_running_sniffer_operating_ipv6_traffic - Looks for processes sniffing all IPv6 traffic.

process_running_sniffer_operating_unknown_protocol - Looks for processes that are sniffing unknown protocol traffic.

process_running_sniffer_operating_all_traffic (disabled by default) - Looks for processes sniffing all network traffic.

A process sniffing traffic is always a high risk activity, but a process sniffing traffic that has suspicious or tampered environments is a red flag. BPFDoor Variants 1 and 2 both have process environments indicative of trying to hide with anti-forensics. We have added the following new checks which will expose BPFDoor variants that are operating on a system.

process_running_sniffer_environ_empty - Looks for sniffer processes with an empty process environment.

process_running_sniffer_environ_corrupt - Looks for sniffer processes with a corrupt process environment.

process_running_sniffer_cmdline_overwrite - Looks for sniffer processes with a command line that was likely overwritten to conceal its real name.

BPFDoor Detection White Paper

We wrote up a full white paper on detecting BPFDoor with Sandfly that is available in our White Papers section. We advise customers concerned with this threat to read the paper as it goes into all the detection features available in Sandfly to find this threat agentlessly and automatically.

Sandfly White Papers

Trickle Scan and Performance Improvements

Finally, we have added in a new “trickle scan” mode. Trickle scan is part of the scheduler where we will slowly spread out scans over the scheduled interval vs. doing them all at once like we have in the past. By tricking out scans, it consumes less network bandwidth and lowers potential host impacts in Virtual Machine server environments. Scans still maintain their random nature for evasion resistance, they will be run in a distributed way.

The trickle scan feature is enabled by default for all scans. If you want the traditional method of doing all scans at once when run, you can set the flag in the scan configuration when adding or editing a schedule..

Upgrading Sandfly

Sandfly 5.5 has many important new features. We encourage customers to try out the new AI LLM analyst and report back how we can improve it. The new BPFDoor features are also important for customers, especially those in critical infrastructure. We strongly encourage an upgrade.

Please be sure to see our documentation on the new features and capabilities:

Sandfly Documentation

Customers wishing to upgrade can follow the instructions here:

Upgrading Sandfly

If you have any questions, please reach out to us.

Thank you for using Sandfly.