With the prevalence of targeted "APT-style" attacks and the business risks of data breaches reaching the board level, the market for "security appliances" is as hot as it has ever been. Many organisations feel the need to beef up their security - and vendors of security appliances offer a plethora of content-inspection / email-security / anti-APT appliances, along with glossy marketing brochures full of impressive-sounding claims.

Decisionmakers often compare the offerings on criteria such as easy integration with existing systems, manageability, false-positive-rate etc. Unfortunately, they often don't have enough data to answer the question "will installing this appliance make my network more or less secure?".

Most security appliances are Linux-based, and use a rather large number of open-source libraries to parse the untrusted data stream which they are inspecting. These libraries, along with the proprietary code by the vendor, form the "attack surface" of the appliance, e.g. the code that is exposed to an outside attacker looking to attack the appliance. All security appliances require a privileged position on the network - a position where all or most incoming and outgoing traffic can be seen. This means that vulnerabilities within security appliances give an attacker a particularly privileged position - and implies that the security of the appliance itself is rather important.

Installing an insecure appliance will make your network less secure instead of safer. If best engineering practices are not followed by the vendor, a mistake in any of the libraries parsing the incoming data will compromise the entire appliance.

How can you decide whether an appliance is secure or not? Performing an in-depth third-party security assessment of the appliance may be impractical for financial, legal, and organisational reasons.

Five questions to ask the vendor of a security appliance

In the absence of such an assessment, there are a few questions you should ask the vendor prior to making a purchasing decision:

1. What third-party libraries interact directly with the incoming data, and what are the processes to react to security issues published in these libraries?
2. Are all these third-party libraries sandboxed in a sandbox that is recognized as industry-standard? The sandbox Google uses in Chrome and Adobe uses in Acrobat Reader is open-source and has undergone a lot of scrutiny, so have the isolation features of KVM and qemu. Are any third-party libraries running outside of a sandbox or an internal virtualization environment? If so, why, and what is the timeline to address this?
3. How much of the proprietary code which directly interacts with the incoming data runs outside of a sandbox? To what extent has this code been security-reviewed?
4. Is the vendor willing to provide a hard disk image for a basic assessment by a third-party security consultancy? Misconfigured permissions that allow privilege escalation happen all-too often, so basic permissions lockdown should have happened on the appliance.
5. In the case of a breach in your company, what is the process through which your forensics team can acquire memory images and hard disk images from the appliance?

A vendor that takes their product quality (and hence your data security) seriously will be able to answer these questions, and will be able to confidently state that all third-party parsers and a large fraction of their proprietary code runs sandboxed or virtualized, and that the configuration of the machine has been reasonably locked down - and will be willing to provide evidence for this (for example a disk image or virtual appliance along with permission to inspect).

Why am I qualified to write this?

From 2004 to 2011 I was CEO of a security company called zynamics that was acquired by Google in 2011. Among other things, we used to sell a security appliance that inspected untrusted malware. I know the technical side involved with building such an appliance, and I understand the business needs of both customers and vendors. I also know quite a bit about the process of finding and exploiting vulnerabilities, having worked in that area since 2000.

Our appliance at the time was Debian-based - and the complex processing of incoming malware happened inside either memory-safe languages or inside a locked-down virtualized environment (emulator), inside a reasonably locked-down Linux machine. This does not mean that we never had security issues (we had XSS problems at one point where strings extracted from the malware could be used to inject into the Web UI etc.) - but we made a reasonable effort to adhere to best engineering practices available to keep the box secure. Security problems happen, but mitigating their impact is not rocket science - good, robust, and free software exists that can sandbox code, and the engineering effort to implement such mitigations is not excessive.

Bonus questions for particularly good vendors

If your vendor can answer the 5 questions above in a satisfactory way, his performance is already head-and-shoulders above the industry average. If you wish to further encourage the vendor to be proactive about your data security, you can ask the following "bonus questions":