In the first two installments of this three-part series based on our recent white paper, The Skeptic’s Guide to Buying Security Tools, we provided an evidence-based approach to helping your organization justify a new security tool purchase and key considerations to future-proof that investment. In today’s post, we’ll discuss how to evaluate the efficacy of that tool once you’ve made the purchase and implemented it within your security ecosystem. 

Can we prove we are safer after the security tool purchase? 

Every technology environment is different. Asking cybersecurity vendors to take part in proof of concept (PoC) demonstrations before buying a product or service became common practice as companies scrutinized new investments more closely. Targeted testing before buying new products or platforms reveals a lot about the solution’s capabilities and its ability to integrate with your current or planned stack.

But, even with reasonable assurances beforehand, the ultimate value of cybersecurity purchases gets decided later, after you’ve deployed the tool and your environment changes again. Before declaring an investment a success or a failure, the skeptical stakeholder asks, “Are we safer today than we were yesterday because we bought this tool?” 

The answer to this question typically impacts future investments, and the reputation of the CISO or team leader who advocated for making the purchase. The only way to avoid having the answer be largely subjective is to be able to quantify and demonstrate value (or the lack thereof) after deploying the solution—sort of like running another PoC to check the first one.

Evaluating technology that’s already deployed requires a reliable and objective means of continuous testing—measurement—throughout the life cycle of investments. As the heart of this ongoing assessment, breach and attack simulation (BAS) lets you automate and model real-world attack scenarios against production environments to validate that controls are in place, properly configured, and working as advertised. 

Ongoing quantification helps maximize—and justify—investments in security posture as time goes on. 

BAS helps to answer the tough questions so those controlling the budget will be more inclined to invest wisely the next time you tell them it’s a good idea. What are the tough questions? Read on to learn more.

Are we better at prioritizing?

Modern security infrastructures include perimeter defenses along with detection and response tools and intelligence and analytics solutions. All of these specialized tools add valuable insight and detail,  but the complexity created by countless handoffs and integrations between tools drains cycles and may actually slow the process of investigating threats—the exact opposite of what we hope to gain from buying tools in the first place. Even with large budgets and well-developed security operations centers (SOCs), most teams lack the cycles, skills, or funds to find and prioritize every gap identified by every tool, until it’s too late. 

One crucial determinant of a tool’s value is whether it makes it harder or easier to find and prioritize threats. To measure whether a tool or platform reduced response times, you need to first have a solid security baseline. BAS can help establish that baseline by running attack simulations against the tools and controls you have in place, then aggregating simulation results into easily digestible dashboards and reports.

You can then rerun previous threats and attacks anytime a new tool is added to measure whether your team is able to flag them faster, helping to evaluate mean time to detect (MTTD) and mean time to response (MTTR) as your security controls change across your entire ecosystem. As your business and threat landscape evolve, BAS uniquely lets analysts automate testing so you can tell whether you’re maintaining your ability to detect and respond to predictable threats.

What happens if we turn on more features on certain firewalls? Does VPN security hold up if our remote traffic increases sharply? If we’re merging with or acquiring another company, which team detects and responds to which threats faster?

With bottom-line facts in hand, the skeptical stakeholder then evaluates the net impact of security purchases on operations and available skills.

Have we empowered or exhausted our team?

Answering this question requires some correlation of speed and efficiency gains, and the presumed reduction of risk, against time and effort spent making things work. Measuring MTTD and MTTR covers one half of this equation. 

But clearly there are times when adding a sophisticated new product or platform requires intensive training, integration, ongoing management and configuration—and is totally worth it. Here, measuring the volume or percentage of false positives and the situations under which they occur adds another key piece of the puzzle. 

Keeping track of the time spent updating the tool, and whether that requires taking tools or segments offline, tells even more. And last but not least, as you evaluate new solutions, BAS helps determine how well the previous purchase integrates with required or potential new additions from the same or different vendors.   

Was it worth the money?

This last—and perhaps most important—question is the skeptical buyer’s dealbreaker and stems in part from sheer mathematics. Research shows cybersecurity budgets continue to rise, but there’s rarely enough money to do everything CISOs and their teams would like to do with unlimited funds. 

Along with measuring the overall impact on response times, another key consideration might be whether you are fully utilizing the tool and, if not, why. Underutilized tools take up a lot of space and may require the same number of cycles to update whether you’re using all of the features or not. Even if each tool excels at one thing, the more shelfware piles up or requires more effort to accomplish disparate tasks, the more closely future purchases get scrutinized.

BAS has the ability to uniquely show you whether the solution you recommended:

• Fixed the problem you were looking to solve—and others
• Streamlined workflows and reporting
• Measurably improved your security posture and processes
• Made your team more collaborative and proactive

Lifecycle Validation for a Security Tool Purchase

Are we more prepared to handle the changing threat landscape? 

Even skeptics can’t dispute the need to invest in modernizing their security stack to stay a step ahead of advanced threat actors. The more risk and security management leaders can continuously demonstrate their ability to clearly identify gaps in security controls and recommend new tools and solutions that effectively close those gaps—with tangible evidence—the more quickly and confidently budget holders within their company will support necessary investments. 

The key word here is “continuous.” Gartner and other thought leaders recognize the need for ongoing validation of security ecosystems and the tools within them. They also recognize the unique role that BAS plays in that type of continuous validation, both as part of a continuous security validation (CSV) program—an automated approach that uses security tools and techniques leveraging attacker TTPs—or a continuous threat exposure management (CTEM) program—a programmatic approach to reducing risk. 

These and other frameworks pull together many disparate elements and point solutions, each of which must be validated on its own merit and on how well it integrates into your unique environment. The more mature your security stack becomes, the more it pays to be skeptical and put prospective investments to the test before making decisions. 

To learn more about the skeptical buyer’s approach, download the complete Skeptic’s Guide to Buying Security Tools white paper. Interested in putting your skeptical buyer skills to the test to see if BAS might be the right tool for you? Check out our Four Pillars of BAS white paper or schedule a demo with a SafeBreach cybersecurity expert.