Drive Effective Governance of AI and Cybersecurity Risk with Effective Performance Measures

There is an academic difference between metrics and performance measurement. The goal of this presentation is to explain how board directors and c-suite leaders can drive performance and demonstrate value by communicating the right information to the right people in the right format at the right time.

The process starts with metrics; however, good metrics are not the point the ultimate goal. “A metric is some data and an algorithm for reducing it to tell a story. The significance of a metric comes from its context. It is specific information about relevant processes" (Ranum). Metrics are ineffective by themselves - especially when you apply them something complex like to emerging technology, generative AI, or something that is a maturing business function like cybersecurity.

Personal Example: My wife and I have been together for 245,000 hours. This number is too large to be relevant to most people receiving the information. The metric drives performance based on the relevant data that makes the number mean something. I have a good idea about what my wife thinks about me and my actions and how much she loves me based on her responses to my actions. This information is not available simply as a result of the time that we have been together. 

Practical Example: There are many questions we can ask that help drive good outcomes for AI and security in modern businesses.

• Who has access to data?
• What can they access?
• When can they access it?
• From where can they access the data?
• For what purpose?
• How do you provide that the data is only accessed in an authorized way by authorized people at an authorized time?
• Are my tools effective to produce the outcomes that I desire? 

Measuring the right things.

The things we measure most are, Length, Mass, Time, and Temperature. These measurements help people understand the world around us. We have no problem measuring tangible items. Measuring intangible items is more difficult.

The standard of measurement is important to ensure we avoid confusion. Standards allow everyone in the conversation to understand what is being measured and the context of the results.

• Meter versus foot
• Kilogram versus pound
• Celsius versus Fahrenheit 

Performance Measures are usually measurements with specific purpose and intention. Much of what security does is hard to quantify, so metrics that evaluate the right information in the right format for the right people to serve the right purpose are important. Most often, effective security metrics highlight impacts to create a meaningful discussion about the most important question: "SO WHAT"

• Does this demonstrate that an investment was valuable?
• Does this highlight that we are using the right services and solutions?
• Does this demonstrate that the needs of the business are being met?
• Does this demonstrate that we selected the right strategy, and we are executing it properly.

Enterprise Risk Performance. If we go beyond security metrics and focus on management of enterprise risk, we have even more questions.

Given what we are measuring, we must know:

• Is it material?
• Is it essential to the operations or promises of the business?
• Do the right people understand the results of our measurements?

What you measure and how you measure it are equally important. 

Are we doing what we said we wanted to do? Really? Prove it! How are you measuring? Qualitatively? Quantitatively?

Resource: NIST SP 800-55 describes the qualities of good metrics: quantifiable; repeatable; readily obtainable; and useful. The attributes in NIST 800-55 align with the purpose of identifying poor performance and pointing to appropriate corrective action using three distinct types of measures.

Once good metrics are established, they need to be categorized:

• Implementation measures to measure the execution of security programs
• Effectiveness/efficiency measures to measure results of security services delivery
• Impact measures to measure business or mission consequences of security events.

How to get started. I use forensic science to enhance what I have selected from NIST 800-55 to ensure what I present to stakeholders is going to be meaningful. There are five formal forensic science activities. They work to produce the information used to address forensic questions about trace evidence, where trace evidence is anything that you can observe. Authentication is the activity that matters. In forensic science, authentication is a decision process attempting to establish sufficient confidence in the truth of some claim. This helps drive the value and Impact of measures communicated to stakeholders. 

Maturity models must die!

Focus on impact to critical systems. If it is important, it is material. For example, the SEC cybersecurity rule made maturity models obsolete. Maturity models are so notional that litigators are going to have a field day.

Keeping it Real: What matters?

There are only three things that you can do with information that is presented to you:

1. Accept the information. 
2. Reject the information.
3. Seek to understand the information.

The effort to understand information and drive meaningful action requires an inventory of what matters to the organization (it is material if it matters). Everything that matters must exist and operate within defined risk boundaries. Once established, the organization can apply the legal definition of the duty of care to ensure proper steps have been taken to operate within the acceptable boundaries of risk taking. The duty of care ensures the organization understands the interests of all parties that might be harmed by the risk. This drives reasonable organizations to take action to intentionally reduce risk such that harms to stakeholders do not result in remedies directed by regulators or the courts.

Executive Summary

Prioritize performance measures based on business impact to ensure appropriate action is taken. The universe of possible metrics can be very large. You want to start small and build upon your success. Standards help everyone to speak the same language. Metrics to paint a clear picture of the situation. Focus on problems that drive action and decisions. Present findings in the context of the business.

✝️ Peace be with you.