I’ve always been a believer in the saying, “If you can measure it, you can manage it!” Metrics seem to be first thing security professionals think of, but typically the last thing to be implemented, and understandably so because you need a process in place before you can start measuring.
I propose a change in how metrics are perceived. Most people explicitly measure the positive, not the negative. For example, it’s easy for executives to agree on the success of patching when you report that server patching is 80% effective. But the inverse of that metric equates to 20% of servers that aren’t patched. Regardless of the percentage reported, the percentage patched in the negative solicits a much different perspective on that metric. In other words, an executive who see a positive metric rarely expects to see 100%; however, when a negative metric is presented there’s pressure to move that to 0%.
The second thing I propose is that metrics should be tied to business objectives. Metrics should articulate strategic alignment with a business driver. As a security department you’ll have much better success at budget negotiation time when you can clearly demonstrate that security initiatives support the overall business strategy.
The third thing I propose is to purposefully structure the context of the metric. This means you need to know the business objectives, high-value services or assets, critical security controls, critical business risks, or disruptive events that could impact the brand value of the company or the hard-earned revenue stream. Once you’ve considered these areas, you need to think about how the metric is going to be composed (i.e., is it projects, tasks, performance goals, fiscal investment?). You should be prepared to explain how the metric is composed and why. If there’s any doubt to the accuracy or completeness of the metric, you’ll lose credibility.
For consistency in the metric examples, the scenario we’ll use is the outsourcing of security operations. Remember, these are strategic metrics, and if measured correctly will lead to good conversation.
Top 10 Metrics (from the folks at the Carnegie Mellon University CERT)
- Percentage of security activities that DO NOT support the business objectives. This could be project count, SLA on the part of the outsourcer, retraining of staff, number of security processes that remain in-house, etc.
- Number of security activities/projects tied to business objectives. You want to be equal to or greater than one, here. Alignment with the CEO or other executive stakeholder goals would be a good place to start gathering business objectives. Feel this out a bit and report in the positive or negative based on the response you’re looking to receive.
- Percentage of high-value services that DO NOT satisfy the security requirements. One example of the outsourcer scenario above, this metric could be a performance review of your outsourcer to confirm compliance. If there are issues reported, you should be prepared to speak to the corrective actions taken.
- Percentage of high-value assets that DO NOT satisfy their security requirements. One example might be network access control lists without business justification. Another example is incidents spawning from incomplete systems configurations.
- Percentage of high-value services with controls that are ineffective or inadequate. This could be an extension of metric #3 to extend to the number ineffective controls identified in the service, or noncompliance with specified controls in the outsourced SLA.
- Percentage of high-value assets with controls that are ineffective or inadequate. This is the same as #5, only the metric applies to assets instead of services.
- Confidence that all risks that need to be identified have been identified. This one is my favorite, best described in a screen shot from the Carnegie Mellon University CERT.
- Percentage of risks with impact above threshold. Examples include risks without mitigation plans (should be 0), or risks that are effectively mitigated by their mitigation plans (should be 100%).
- Probability of delivered service throughout a disruptive event. You can get creative with this metric, but some angles to consider include the probability of delivering high-value controls in crisis mode, or results from outsourced normal SLAs given disrupted operations.
- For disrupted, high-value services with a service continuity (SC) plan, percentage of services that did not deliver service as-intended throughout the disruptive event (i.e., services with SC plans that do not maintain required service levels identified).
Included Topics
