Week 44: Cybersecurity Metrics

In the last 43 weeks, we’ve covered the theory behind topics like, but not limited to, Risk Management, Incident Response, and Cyber Threat Intelligence. And, all of that is fine, but we felt something something was missing – Metrics. 

 So, why are they crucial?

Metrics can help organizations evaluate the effectiveness of their security efforts against threats. Tracking them can provide a view of the events that occurred in the last x number of days, for example, and where they materialized. They can provide intel into the robustness of security tools and processes. They can help measure aspects of incident response, vulnerability management, and compliance. All this means that security teams can track, analyze, and enhance their approach towards security. It can even aid CISOs to prioritize investments. 

As a recap, measuring the effectiveness of cybersecurity with tangible data can:

• Help organizations to be more proactive by pinpointing weaknesses in their security posture. 
• Answer questions like, “How can we improve our security practices?” and “Are we investing adequately?”
• Aid in measuring progress towards organizational goals around security as well as compliance requirements (laws and regulations, as well as standards like ISO 27001 and PCI DSS). 
• Provide data with regard to competitors and staffing needs (incl. awareness, training, and education). 
• Improve resilience and reduce the risk of future attacks.

Okay, but why do entities fail to measure their cybersecurity effectiveness?

According to an article published by CIO in 2017, Thycotic, a provider of privileged account management (PAM), conducted a survey in which they interviewed over 400 businesses. The study revealed that companies spend over $100 billion on cybersecurity defenses. Approximately, 32% made business decisions and purchased cybersecurity technology blindly. Over 80% did not consider the business users when making decisions about procurement. The article also said that these companies did not have a steering committee to evaluate the risks and impacts with regard to the investments. 

Additionally, the ISF found that many CISOs report the wrong metrics. This is because they have little to no interaction with the intended audience. Consequently, plenty of assumptions are made, which leads to a disjoint between security and the business. 

(Note: In our experience, too many metrics can also affect decision making). 

Hmm… what does all this translate to?

It means the following process should be followed:

• It's important to understand the business and security goals.
• Identify the key stakeholders.
• Design KPIs/ KRIs.
• Report regularly. 
• Evolve as the goals shift.  

(Note: Key Performance Indicators (KPIs) and Key Risk Indicators (KRIs) are two different metrics. KPIs focus on analyzing the overall performance of the organization, while the latter is concerned with identifying and keeping track of risks that could affect the organization’s ability to achieve its objectives). 

Question: “Is there a framework to measure your cybersecurity program?”

Yes, and it’s called CARE. It was designed by Gartner, and it stands for Consistent, Adequate, Reasonable, and Effective.

• Consistency: This assesses whether the control has been working consistently over time. This should be regularly measured, updated, and reported (whether weekly, monthly, or quarterly). One example of security awareness could be the percentage of employees who have received training around phishing in the past X months.
• Adequacy: This answers the question, “Is the control meeting the needs of the business and the expectations of the different stakeholders?” For example, with regards to patching: Percentage of assets patched with a protection-level agreement (PLA). 
• Reasonableness: This analyzes if the controls are appropriate with regards to the business impact and the friction caused. One example of delay and downtime could be the average delay while creating a new access. 
• Effectiveness: This ensures that your controls are producing the desired outcome. For instance, with regards to the cloud, the number of issues around cloud security per year that are related to configuration. 

In a nutshell, this framework, unlike the traditional approach where organizations measure the avoidance of an outcome, CARE helps assess the credibility and defensibility of the cybersecurity initiative. 

Now… where can I find examples? 

We couldn’t find an exhaustive and/ or downloadable list, but we did find resources like SecurityScorecard that mention some of the metrics you can track and share with the relevant stakeholders. This pdf from Actzero, for instance, provides an example of how you can structure your document by using columns like audience level, owner, InfoSec. category, KPI, definition, and rationale.  

Next week we’ll address security by design. 

This article is part of a project called Security Chronicles, written jointly with Walter Buyu .

Sources:

• Decoding Cybersecurity Metrics: Top 10 KPIs Every CISO Must Know.
• SOC Metrics: Security Metrics and KPIs for Measuring SOC Success. 
• How to measure cybersecurity effectiveness – before it’s too late. 
• Integrating KRIs and KPIs for Effective Technology Risk Management.
• 5 Strategies for Setting the Right Cybersecurity KPIs.
• 4 Metrics That Prove Your Cybersecurity Program Works.