Tuesday, June 25, 2013

Ten Dimensions of Cyber Security Performance

I am proposing a framework for managing cyber security performance.  As a preview, here are the ten dimensions:
  1. Optimize Exposure: attack surface and vulnerabilities, including assets, people, processes, & technologies
  2. Effective Threat Intelligence: understanding the threat agents 
  3. Effective Design & Development: security & privacy by design 
  4. Quality of Protection & Controls
  5. Effective/Efficient Execution & Operations 
  6. Effective Response, Recovery, & Resilience
  7. Effective External Engagement: responsibilities and risk drivers
  8. Effective Learning & Agility: OODA at an organization level
  9. Optimize Total Cost of Risk: (loss distribution approach)
  10. Responsibility & Accountability: including governance and compliance
Each of the ten dimensions are explored in subsequent posts (see links above).  The interactions among the first six dimensions are discussed in a post called "Operational Cyber Security & Single Loop Learning".  The interactions among the second four dimensions are discussed in "Agile Cyber Security and Double Loop Learning".

Here is are the slides of the diagram, built dimension by dimension.

These posts might be especially interesting to folks who engaged in any of the Cyber Security Framework processes now underway in the US (NIST), EU, or the UK.

(Comment: I acknowledge that this framework is a bit complicated.  My friend Jack Whitsitt   (@sintixerr) has suggested that we need "crayon version" because anything more complicated will just confuse people.  I concur, but I may have to flesh out this complicated version before I can get to something as simple as Plan-Do-Check-Act.)"


The aim is to promote agility, encourage capability maturity, and promote rapid innovation across the ecosystem. It is designed for managers, executives, and governance bodies. 


This framework is based on the Balanced Scorecard concept, augmented with ideas from Enterprise Risk Management, Total Quality Management, Organization Science, and even Biology (e.g. human immune systems).


By "cyber security" I mean the confluence of information security, industrial control security, privacy, identity, and digital rights, along with civil liberties and national/homeland security in the digital domain. (I need an umbrella term and this is the best I can find.)

By "performance" I mean results that can accomplished that are (mostly) under defender's control, even in the face of a rapidly-evolving landscape of threats, technologies, and socio-economic-political conditions. While "outcomes" will be determined by the stochastic processes and strategic behavior of adversaries, I argue that "performance" can and should be the focus of management.

"Performance" is not merely the sum of cyber security activities executed in an organization.  In my opinion, existing frameworks focus too much on tasks, activities and practices -- e.g. "identify threats", "keep software patches up to date", "prioritize controls based on risk assessment", etc.  This can lead to a belief that cyber security is merely the accumulation of activities and that "good cyber security" is some static goal or state.  This belief is wrong-headed if our goal is agility and innovation.  What's missing from this static view is the "why" for each task, activity, and practice, and especially how they all work together to promote agility, capability maturity, and innovation.  Here's my proposed definition:
  • "Cyber security performance" -- systematic improvements in an organization's dynamic posture or capabilities relative to its rapidly-changing and uncertain adversarial environment.
Consider the case of vulnerability patching/fixing.  By itself, I'd call it a set of activities and practices.  Let's say an organization found and fixed 15 vulnerabilities in a month.  That is clearly activity, but is that "performance"?  Most teams rank and triage vulnerabilities based on "criticality" and maybe also age, because fixing the most critical vulnerabilities is thought to contribute most to improved informations security.

But I would argue that, alone, finding and fixing vulnerabilities do not yield "systematic improvements" as per the definition above.  What about the sources and root causes of those vulnerabilities?  What about vulnerabilities that have not yet been discovered? And how should vulnerability finding and fixing be balanced against other tactics and strategies, e.g. "moving target" defenses?   As Peter Drucker said: 
"Efficiency is about doing things right.  Effectiveness is about doing the right things".
"There is nothing more wasteful than becoming highly efficient at doing the wrong thing." 
Therefore, every activity related to cyber security needs to be evaluated against its overall effectiveness. To aid in this, I think its useful to distinguish between capabilities and associated performance dimensions.

Measuring Performance Across Ten Dimensions

Cyber security performance is a characteristic of the system as a whole.  Capabilities interact with capabilities systematically to deliver cyber security performance.  Therefore, if we can define the general capabilities we need and how they interact, we should be able to define and measure cyber security performance.

As listed above, I'm proposing ten capabilities which can be measured as performance dimensions.  Why ten?  Because I think cyber security is too complex to be reduced to any fewer dimensions, and I can't see the value of adding more with finer distinctions.  I'm arguing that these ten dimensions are both necessary and sufficient to manage the full scope of cyber security in nearly every organization or network of organizations.  In simple language, you can't leave out any and these are all you need.

Again, each of the ten dimensions will be explored in subsequent posts.

Finally, I will be proposing a performance index each dimension.  It will use statistical inference to combine evidence rather than simple arithmetic to combine sub-metrics.  I'll explain this method in a separate blog post.

(Edit 7/1/2013: Modified order of ten dimensions, plus modified a few titles. Still a draft.)


  1. Fascinating work. There's a lot to take in here, but I can tell right way that we are in sync on designing for correct protection levels and the business agility that can be attained. I'm not sure threat exposure can every be fully optimized but the end goal is a continued pilgrimage toward optimization. They key in my mind is for people to understand that the journey has to begin. Instead I see professionals look for the easy button through addiction to frameworks, expensive technology, and buzzword bingo.

    What I'm finding infinitely fascinating is that we appear to be approaching similar conclusions from entirely different starting points (there must be a scientific term for that). One area that might be different is in value to the organization. In my corporate role decisions are based around using security to help build value to business processes. But I also see that as implicit in the 10 dimensions so its probably just a difference in the environmental influence we both live in. Oddly for me a strong base of this came from adapting concepts from Saddle Back Church's Purpose Driven principles (removing the theology its an interesting work of lean manufacturing principles).

    For me the concept of value has to be dealt with to ensure that risk treated correctly. We will take a lot more steps to treat risk to our family than to the tomato plant in the back yard I guess. Protecting the most sensitive and valuable aspects of organizations; those things than will prevent them from doing what they do. Your measurement of cost is great in that respect. I'm curious if your 10 dimensions draw from Six Sigma DMAIC in some way?

    That's all just off the top of my head. There's a lot here and it will take me time to digest. My mind is more of a rock tumbler than quick strike surgical tool :)

  2. Thanks for your comments, Dave. Other readers should read Dave's blog post a reference: http://orthosec.blogspot.it/2013/07/risk-management-value-oriented-process.html.

    Regarding "value to the organization", I think we agree. I didn't call this out explicitly in the 10 Dimensions, partly to keep the presentation shorter and focused. But let me say it clearly now (and maybe also in a separate post): cyber security exists to support the value-creating processes of an organization, and also the value-creating processes of stakeholders. This relationship is captured in the financial services industry in their use of the measure "Risk-adjusted Return on Capital" (RAROC).

    Yes, the 10 dimensions draw on ideas from 6 Sigma and DMAIC in several places. But it's not a full transplant since there are some very important differences from quality improvement in manufacturing or service.

  3. Just completed a quick scan of this post and the 10 supporting posts. Very interesting concepts especially since they are worked together so completely. It will take some digestion to really see where I agree and disagree.
    Big question for me is how this can/will apply to the industrial control system field as opposed to IT. At first glance it should apply without to much modifification.

  4. Superb stuff Russell ... all of it. But let's extend:

    "What about the sources and root causes of those vulnerabilities? What about vulnerabilities that have not yet been discovered?"

    What about backing up even further and making business decisions (e.g., M&A, strategic partnering, new system acquisition and deployment, hiring, etc.) with "new vulnerability minimization" having a seat at the table? ab

    1. Thanks, Andy.

      > What about backing up even further and making business decisions

      You are sing my hymn, brother! That is exactly the sort of change that the Ten Dimensions is aimed at promoting. Yet I know that for the vast majority of organizations, this would require a huge shift in culture, practices, and even hiring/promotion within InfoSec and related groups. To get there would take some serious vision and commitment. But that's where I think we need to be going.