Review of Whitsitt's "B-side Cyber Security Framework" (Mapped to the Ten Dimensions)

My colleague Jack Whitsitt (@sintixerr) has proposed a B-side version of the NIST Cyber Security Framework (NIST CSF) in this blog post.  In this post I will give my comments on Jack's framework, and do so by mapping it to the Ten Dimensions.

The NIST CSF is a catalog of information security practices, organized into categories and maturity tiers. I've criticized the NIST-CSF here, here, and here, and proposed an alternative -- the Ten Dimensions.  Jack has posted commentary and critiques here, here and  here.  Jack has the advantage of participating in all five workshops, plus several side meetings with various players.

Here's a diagram of Jack's framework:

Short Summary

I like Jack's B-sides framework. I see a lot of overlap between it and my Ten Dimensions.  They aren't identical but the same themes come through in both. His has the advantage of simpler interpretation (top-down layer cake, half as many dimensions).  It has short-comings as well.  In it's current form, it lacks performance measurement and, in my opinion, as inadequate attention to "Effective Response, Recovery, & Resilience", "Effective External Engagement", "Optimize Cost of Risk", and organization learning loops.

Mr Langner is wrong. Risk management isn't 'bound to fail'. But it does need improvement and innovation.

In "Bound to Fail: Why Cyber Security Risk Cannot Simply Be 'Managed' Away" (Feb 2013) and a recent white paper, Ralph Langer argues that risk management is a fundamentally flawed approach to cyber security, especially for critical infrastructure.

Langner's views have persuaded some people and received attention in the media.  He gained some fame in the course of the investigation of the Stuxnet worm capabilities to exploit Siemens PLCs (programmable logic controllers). Specifically, Ralph was the first to assert that Stuxnet worm is a precision weapon aimed at sabotaging Iran's nuclear program. Langner also gains institutional credibility as a Nonresident Fellow at the Brookings Institute, who published the "Bound to Fail..." paper.  I'm guessing that Brookings PR department has been helping to get press attention for Langner's blog post critiquing NIST CSF and his proposed alternative: RIPE.  They were reported in seven on-line publications last week alone: here, here, here, here, here, here, and here.   (Note to self: get a publicist.)

In this long post, I'm going to critique Mr. Langner's critique of risk management, pointing to a few places where I agree with him, but I will present counter-arguments to his arguments that risk management is fundamentally flawed.

• TL;DR version: There's plenty of innovation potential in the modern approach to risk management that Langner hasn't considered or doesn't know about. Therefore, "bound to fail" is false.  Instead, things are just now getting interesting.  Invest more, not less.

In the next post, I'll critique Mr. Langner's proposed alternative for an industrial control system security framework, which he dubs "Robust ICS Planning and Evaluation" (RIPE).

Latest #NISTCSF draft: "Three yards and a cloud of dust" will not be enough to matter

I wish I could write a more favorable review of the latest NIST Cyber Security Framework (CSF) draft.  I'm in favor of frameworks that might help us break out of the current malaise in cyber security.  I'm not anti-government or anti-regulation, either.  But my review isn't favorable because I don't think NIST CSF will promote the type of change necessary to make a meaningful difference.  It's not about the details but mostly about the overall structure and strategy behind it, as I describe below.

CSF advances the ball, but not enough to matter,
especially considering the effort.

The title of this blog post refers to the offensive strategy of Ohio State football coach Woody Hayes.  He had success for nearly 30 years with a very conservative, grind-it-out ground-based offense that centered on running plays between the Tackles. His teams often threw fewer than 10 passes a game.  Since a team needs 10 yards every 4 plays to hold on to the ball, if you could guarantee 3 yards per play you could hold on to the ball for a long time and eventually grind it in for a score.  This also worked in part it minimized turnovers.  It was predictable and not very imaginative, but succeeded through sheer mass, strength, effort, and persistence.

This may have worked for Woody at Ohio State, but it's a poor model for progress in cyber security.  The "pace of the game" is governed by the clock speed of innovation on the part of threat agents, not by us defenders.  Plus, most organizations aren't sitting at "3rd down and 3 yds to go" -- more like "3rd down and 33 yds to go".  "Three yards and a cloud of dust" is ultimately a failing strategy because it leaves us perpetually behind the minimum threshold of acceptable performance.

First Presentation of "Ten Dimensions..." at BSides-LA

I had fun on Friday presenting the "Ten Dimensions of Cyber Security Performance" at BSides-LA.  This is the first time I presented it in a general forum, so I was looking forward to see how it would "fly" and to see what reactions it would get.

On the plus side, several people were pretty excited and I had some great discussions afterward.  Also, I got most of the presentation done in the available time, but I still have more tuning to do.

On the down side, there weren't as many people in my session as I had hoped.  It was one of the last sessions on the last day, so that probably had an impact.  Or maybe the headline or topic wasn't widely interesting.  But the people who were there were interested and engaged, which is what matters most.

But for a first presentation, I felt it was successful.

Here are the slides.  View in full screen mode to enjoy the animations.

How simple can we make it?

In the Q&A session of the first day of the 3rd NIST Cybersecurity Framework (CSF) workshop, someone asked if there was a way to simplify the proposed five functional categories in the Core.  Basically, he was saying that he needed to persuade non-specialists, especially executives, and that the five functional categories plus subcategories was too complicated. (full question and answer is on the video at 1:18:00).  When I heard that, I nearly sprayed coffee all over my keyboard.  "You want it even SIMPLER??" I yelled out (to my screen).

I immediately thought of this: cyber security in one dimension using the Grissom Dimension, which is named after Astronaut Gus Grissom.  Grissom gave a speech in 1959 to the workers at the Convair plant where the Atlas rocket booster was being built.  The entire speech:  "Do good work." (remembered by a worker)  Yes, we could reduce all of cyber security to the Grissom Dimension, then it would be simple, right?

I'm a bit sensitive to this because I know many people will say my Ten Dimensions are too complicated.  I wonder myself if it is too complicated and I'm certainly interested in ways to simplify it.  Parsimony is good.  Occam's razor keeps our models clean.

Slides for the Ten Dimensional Framework, step by step

Here's the slide deck that builds the block diagram for the Ten Dimensions of Cyber Security Performance.

Recommendations for NIST Cyber Security Framework (CSF) Workshop 3

Here is my input to the NIST CSF process prior to Workshop 3 (July 10-12 in San Diego).
________________________________________________________________________
Dear workshop attendees:

I hope that you find the following recommendations to be helpful.  Wish I could be there!

Russ
________________________________________________________________________

Recommendations:

1. In my judgement, the five "Cyber Security Functions" described in the July 1 draft are inadequate to support agile and continuously innovative cyber security.  As detailed in this analysis, the five functional categories have serious deficiencies:
• "Know" is too broad and too vague
• "Respond" and "Recover" are too narrow and could be combined
• "Detect" does not adequately cover of Threat Intelligence
• Missing:
• Design & Development
• Resilience
• Execution & Operations
• External Engagement
• Agility & Learning
• Total Cost of Risk
• Responsibility & Accountability
2. Rather than using functional categories which are nothing more than "buckets of content", it would be better to organize the framework around performance dimensions.  This will help make the framework more coherent and better justified.
3. I recommend organizing the framework according to Ten Dimensions of Cyber Security Performance, (slides), which are explained individually the linked posts:
1. Optimize Exposure
2. Effective Threat Intelligence
3. Effective Design & Development
4. Quality of Protection & Controls
5. Effective/Efficient Execution & Operations 
6. Effective Response, Recovery, & Resilience
7. Effective External Engagement
8. Effective Learning & Agility
9. Optimize Total Cost of Risk
10. Responsibility & Accountability
4. I recommend that the framework should explicitly support Double Loop Learning, which is described in these two posts:
• Operational Cyber Security & Single Loop Learning
• Agile Cyber Security and Double Loop Learning
5. I recommend that pilot projects be started right away to design and test inference methods for estimating cyber security performance, as sketched in this post.

The key to measuring cyber security: inferences, not calculations

(It's getting very late here, so my apologies for any slips or gaps in communication.)

In many of the previous posts on the Ten Dimensions of Cyber Security Performance, I've hinted or suggested that these could be measured as a performance index.  But I'm sure many readers have frustrated because I haven't spelled out any details or given examples.  Still other readers will be skeptical that this can be done at all.

Sorry about that.  There's only so much I can put in each post without them becoming book chapters. In this post, I'll describe an approach and general method that I believe will work for all or nearly all the performance dimensions.  At the time of writing, this is in the idea stage, and thus it needs to be tested and proven in practice.  Still, I suggest it's worthy of consideration.

[Update 7/19/2013:  After some searching in Google Scholar, I discovered that the method I'm suggesting below is called Bayesian Structural Equation Modeling.  I'm very glad to see that it is an established method that has both substantial theory and software tool support.  I hope to start exploring it in the near future.  I'll post my results as I go.]

Here are my ideas on how to measure performance in each dimension.

NIST's "Cyber Security Functions" compared to the Ten Dimensions

On July 1, NIST posted a draft outline of the CSF.  It proposed five "cyber security functions" to serve as organizing categories for the framework.  Quoting from the draft:

• "Know – Gaining the institutional understanding to identify what systems need to be protected, assess priority in light of organizational mission, and manage processes to achieve cost effective risk management goals."
• "Prevent – Categories of management, technical, and operational activities that enable the organization to decide on the appropriate outcome-based actions to ensure adequate protection against threats to business systems that support critical infrastructure components."
• "Detect –Activities that identify (through ongoing monitoring or other means of observation) the presence of undesirable cyber risk events, and the processes to assess the potential impact of those events."
• "Respond – Specific risk management decisions and activities enacted based upon previously implemented planning (from the Prevent function) relative to estimated impact."
• "Recover - Categories of management, technical, and operational activities that restore services that have previously been impaired through an undesirable cybersecurity risk event."

There are several important differences between these five categories and my proposed Ten Dimensions of Cyber Security Performance.  First, NIST is proposing categories of activities and functions to serve as buckets of content.  There's no formal relationship between the categories, at least not stated explicitly.  Second, the NIST categories only partially and imperfectly cover the space of the Ten Dimensions, as shown in this matrix (click to enlarge):

If you believe in the scope and organization of the Ten Dimensions, then the deficiencies of the NIST functional categories become apparent in the comparison:

1. "Know" category is scoped too broadly. It is overloaded and contains too many performance dimensions.  I list five question marks (?) in the matrix because I can't tell if these would be included in "Know" or not.
2. "Respond" and "Recover" categories map to a single performance dimension, implying that they are probably scoped too narrowly.
3. A glaring omission is lack of coverage for Resilience, which is vital for critical infrastructure.
4. Also there's no coverage of dimension 5. Effective/Efficient Execution & Operations, and probably no coverage of five other dimensions: 3. Effective Design & Development; 7. Effective External Engagement; 8. Effective Agility & Learning; 9. Optimize Total Cost of Risk; and 10. Responsibility & Accountability.

Thus, the NIST functional categories put too much attention in one or two areas and not enough in many others.  Most serious, there is no coverage in the second loop of the Double Loop Learning model, which implies that the NIST functional categories are inadequate to support agile and continuously innovative cyber security.

Agile Cyber Security and Double Loop Learning

In this post, I want to summarize dimensions 7 through 10, focusing on their interactions and relationships and how they deliver Double Loop Learning. (See this post for the full list of dimensions.)

Together, dimensions 7 through 10 provide the "dynamic capabilities" of an organization to achieve agility and rapid innovation in the face of constant changes in the landscape.  I mentioned this specifically in the context of dimension 8. Effective Agility & Learning, but the notion of dynamic capabilities extends to subsystem comprised of dimensions 7 -- 10, as well.

Dimension 10: Responsibility & Accountability

This is the tenth and last post defining each of the Ten Dimensions of Cyber Security Performance.  It's also the capstone of all the performance dimensions, tying them together from the perspective of leadership and management.

The dimension of Responsibility & Accountability include all processes that link the decision-makers in an organization (at all levels) with the stakeholders of the organization who are affected by cyber security outcomes, including:

• The Board of Directors
• Shareholders
• Customers
• Employees (as individuals)
• Suppliers, distributors
• Outsource partners
• Regulators
• Legal authorities
• (others)

This dimension includes most of the processes of governance and compliance, at least the interfaces between organization units and the external interfaces.  But I chose not to call it "Governance & Compliance" because those are both formally codified processes and I felt it was important to include some of the less formal and tacit aspects.  This is especially important if we want to encourage wide-spread acceptance of responsibility and accountability beyond the core executives.  In addition, I felt the title "Responsibility & Accountability" focuses attention on performance, not just activity or formal structures.

Dimension 9: Optimize Total Cost of Risk

This is the ninth post defining each of the Ten Dimensions of Cyber Security Performance.  Like dimension 8. Effective Agility & Learning, this dimension applies to performance of the cyber security program as a whole, and therefore I've located it to the side of the other blocks as if it were a higher level.

The dimension of Optimize Total Cost of Risk includes all of the processes that assess and manage cyber security at an enterprise level, specifically in terms of resources (financial and non-financial), liability (potential and realized), risk mitigation (including insurance), and also balancing these against the upside of taking risks (i.e. the benefits of exposing and using information and systems).  Essentially, this is the financial side of risk management at the enterprise level.

Most existing cyber security frameworks either omit or exclude this dimension, or it is viewed merely as the aggregation of risk estimates for each and every possible attack on every asset, and every possible consequence.

In contrast, the approach I'm recommending for this performance dimension starts at the enterprise level and estimates costs associated with cyber security as a whole, both the costs of defenses and the costs of loss events.  It adapts the loss distribution approach (LDA) for Operational Risk (OR) within Enterprise Risk Management (ERM) that has been pioneered and refined in the financial services industry.  This is a fairly complex topic on it's own, so I won't attempt to give a full exposition here.  There are references with further detail, below.

Dimension 8: Effective Agility & Learning

This is the eighth post defining each of the Ten Dimensions of Cyber Security Performance.  Like dimension 7. Effective External Engagement, performance in this dimension shapes and structures operational cyber security as a whole.  On the block diagram, I'm showing this as being with the focal organization but somewhat apart from the others to indicate that is concerned with holistic performance.

The dimension of Effective Agility & Learning includes the processes of reconfiguring, reengineering, redesigning, and radically innovating the cyber security program as a whole.  This is "Second Loop Learning" in the Double Loop Learning model, and is in contrast to the Single Loop Learning for operational cyber security that was discussed in a previous post.  In a later post, I'll discuss how dimensions 7 through 10 come together as Second Loop Learning.  But for now, you can view this dimension as the processes that execute and make operational this outer learning loop.

Including this dimension in our framework is vital for achieving the goals of dynamic and rapidly innovating cyber security.  Existing cyber security frameworks either omit it or subsume it in other categories in a marginal or vague form.

Dimension 7: Effective External Engagement

This is the seventh post defining each of the Ten Dimensions of Cyber Security Performance.  Thus far, I've defined the six dimensions that comprise operational or day-to-day cyber security.  Dimensions 7 through 10 are qualitatively different in that they guide, structure, and set requirements and constraints for operational cyber security but do not directly control it or influence it.  These dimensions are might be more controversial as essential performance dimensions, but I hope to convince you that they are both necessary and also not subsumed in the first six dimensions.

Performance in the dimension of Effective External Engagement means identifying, understanding, negotiating, implementing, monitoring, and improving relationships with organizations and entities external to the focal organization.  "External relationships" includes technical or informational relationships with service providers such as Internet Service Provider (ISP), DNS service providers, registrars, certificate authorities, federated identity service providers, etc.  I can also include the entire supply chain for an organization regarding information technology and communications.  Obviously this include suppliers of information security products and services.

But it would be a mistake to view this dimension as solely a matter of managing technological relationships or supply chain management.  If it was, then it might be subsumed under the other dimensions.  The rest of this post explains what's new.

Operational Cyber Security and Single Loop Learning

In this post, I want to summarize the previous posts and also describe the interactions and relationships among the first six performance dimensions.  (See this post for the full list of dimensions.)

The core of cyber security performance is at the operational level, which includes processes and activities that drive day-to-day results.  Existing cyber security frameworks include many of the performance dimensions I am proposing, but in my opinion they have gaps and inadequacies:

• Resilience is often omitted or treated as a disjoint collection of processes.
• Design & Development is often omitted or is too narrowly defined to include only software design.
• Execution & Operations is often under appreciated and is subsumed under other categories.
• Protection & Controls is often defined too broadly, as if nearly every pro-active cyber security practice could be defined as a new protection process or control.
• Exposure is often treated too narrowly, focusing on identifying information assets and technical vulnerabilities, and not giving sufficient attention to the total "exposure surface" (analogous to attack surface) which includes all aspects of exposure -- people, processes, technology, and information.  Also, many existing frameworks focus on minimizing exposure and minimizing vulnerabilities, and thus ignore the balancing act of promoting access and use of Systems to achieve organization goals.
• Threat Intelligence, in some frameworks, is focused too narrowly on methods of attack (a.k.a. the "threats") and not enough on the actors behind those methods (the "threat agents").  Also, they tend to put too much attention on malicious actors and not enough on actors how are prone to accidents and errors, and also actors who are exploitive-but-legal.
• All together, existing cyber security frameworks tend to be inadequate in how they treat the intersection of information security, privacy, IP protection, industrial control protection, national/homeland security, civil liberties, and digital rights.

The Ten Performance Dimensions that I am proposing aim to remedy these gaps and inadequacies, which I believe is a useful contribution.  But even more important is the dynamic interaction of these dimensions, because it is those interactions that can yield a system that is agile and capable of rapid innovation.

Dimension 6: Effective Response, Recovery & Resilience

This is the sixth post defining each of the Ten Dimensions of Cyber Security Performance.

Unlike the five previous dimensions, the Effective Response, Recovery & Resilience dimension is concerned only with the stream of Events associated with cyber security. These Events include “incidents”, “breaches”, “leaks”, “compromises”, “violations”, “outages”, and the like. The dimension it is most closely associated is 5. Efficient/Effective Execution & Operations. Thus, it appears on the block diagram just below, and next to “Events”. There is a new interaction path with events, because its possible to have cascading events and processes after the initial incident. (See my recent paper on breach impact estimation.)

This performance dimensions includes processes of incident response, digital forensics, business and legal response and recovery (including regulatory processes), etc. It also includes processes and activities designed to promote resilience -- the ability to continue operating even in the face of cyber attacks or cyber-physical events. These processes are covered in many existing cyber security frameworks so I won't dive into details here. But new issues arise when these are considered as a performance dimension and not just a loose collection.

Dimension 5: Effective/Efficient Execution & Operations

This is the fifth post defining each of the Ten Dimensions of Cyber Security Performance.

The dimension of Effective/Efficient Execution & Operations is closely related to the previous one, 4. Quality of Protection & Controls.  It has nearly identical relationships with the other dimensions and thus is portrayed in the block diagram using the same form.  It has a complementary and mutually supporting relationship with protections and controls.  Crudely speaking, the protections and controls are the "brains" while execution and operations are the "brawn" -- i.e. the "engine" that gets work done within the cyber security system and also in the interface between cyber security and every other aspect of the organization.

It also opens a separate interaction path with Actors, who can engage with the related processes as artifacts themselves, not just as they are implemented in the organization.

This dimension includes processes related to implementing and monitoring cyber security, including logging, training, reporting, patching, configuring and updating (e.g. servers, firewall rules, access control rules), and so on.  Using similar reasoning, the software industry has recently developed a similar concept and method under the name "Development Operations" or "DevOps".  Here are some good summaries on DevOps applied to information security:

• DevOps Cookbook
• RSA presentation
• "Integrating Information Security into DevOps"

If anyone doubts the significance of execution and operations, consider the recent case of the Windows Azure outage, which was caused by an operational error regarding expired SSL certificates.  (described here and here).  This outage had significant financial consequences for Microsoft beyond the response and recovery costs because they offered a compensating credits to their customers.

Moving beyond the details of each of the processes, I'll focus on how to view execution and operations, taken as a whole, can be measured and managed as a performance dimension.

Dimension 4: Quality of Protection & Controls

This is the fourth post defining each of the Ten Dimensions of Cyber Security Performance. These are some initial thoughts presented in a sketchy fashion.  All are subject to much refinement, revision, and condensation.

Conceptually, protection and controls intercede between Threats and Exposures, including sometimes directly through modifications to Systems (e.g. system configuration).  It also envelops Design & Development because it takes the design of Systems and the rest of the cyber security program as given, essentially.

There is also a new interaction path with Actors, meaning that people and organizations of all stripes can encounter and interact with protections and controls as artifacts in themselves, and not just as they are implemented to protect Systems.  A prime example would be security and privacy awareness programs.

This box only goes half-way around because the other half is covered by Dimension 5: Effective/Efficient Execution & Operations.

This performance dimensions includes a large portion of what specialized teams do when they implement cyber security.  Examples include access control, identity management, intrusion detection and prevention, exfiltration controls, malware detection and removal, and so on.  These components are widely known and discussed so I won't spend time elaborating them here.  In fact, it sometimes tempting to reduce all cyber security into a set of relevant protections and controls.  This is a mistake, and one of the purposes of this framework is to correct that mistaken thinking.

Dimension 3: Effective Design & Development

This is the third post defining each of the Ten Dimensions of Cyber Security Performance.

Design & Development determines the general characteristics of an organization and its systems relative to cyber security.  Design includes processes of thinking, planning, and formal definition, while Development includes the realization of designs, including refinements and adjustments.

In the block diagram, I've positioned it underneath the "Systems" block because, in a certain sense, it provides the conceptual foundation and architecture of the Systems and how they behave, i.e. how it generates output events in response to inputs and interactions.  Of course this is a simplification since the influence and scope touches nearly every other performance dimension.

While this obviously includes any software and hardware development performed internally, it's very important to note that this dimension include all relevant design and development activities, including:

• business process design and development
• organization design and development
• enterprise architecture
• information and data architecture
• partners relations, including supply chain, distribution chain, and outsource partner relations
• contracts
• governance, both inside and outside the organization
• incentive systems

Note that this definition excludes design and development performed outside the organization.  For example, if an electric utility firm does no internal software or hardware development (not even through contractors), it depends on software and hardware from independent vendors.  Notice that this firm has made a design decision, namely to "buy" rather than "make".  But beyond that, it has no influence over the design and development decisions of its venders.  Therefore, it would need to manage the performance of its vendors through a different performance dimension: 7. External Engagement.

Design & Development is extremely important to cyber security performance because it has the potential to yield systemic improvements and benefits.  Conversely, neglect could lead to persistent and systemic dysfunctions, with the negative consequences showing up in many areas and functions.  This is one of the important lessons from the Total Quality Management movement -- that quality (or lack of it) is often pre-determined at design time rather than at manufacturing time or later.

Dimension 2: Effective Threat Intelligence

This is second in a series of posts describing each of the Ten Dimensions of Cyber Security Performance.

In terms of the framework I'm proposing, Threat Intelligence mediates between Actors and Systems but only in the context of what is exposed, as suggested by it's position in the block diagram.

Performance in this dimension means developing on-going intelligence to these questions:

• Who or what might be a threat to our information or systems?
• In what setting or context?
• What are the capabilities and interests of these threat agents?
• How do threat agents benefit from our information or systems?
• What are the negative consequences to us?