B-Sides LV slides

Here are my slides for today's B-Sides Las Vegas talk (5pm Wednesday).  I'll be demoing the B-Sides SF spreadsheet (see previous post).  A video of the talk will be available on Archive.org in a day or so.

B-Sides SF Talk

Here is the demo spreadsheet I'll be using in today's B-Sides SF talk on the Thomas Scoring System (TSS):

• InfoSec Maturity Assessment (TSS) V1.xlsx

Download the spreadsheet and open in Microsoft Excel 2008 or later.  It uses conditional formatting and cell data validation, but no macros or other advanced features.  The sheets are protected to avoid data entry errors, but there is no password.

This is a realistic, fully functional implementation of the TSS applied to a general case: scoring the maturity of a company's information security capability.

Precision vs Accuracy

When ever you do any kind of measurement, it's important to understand the uncertainties associated with it.  Two characteristics of measurement that are inverse to uncertainties are 'precision' and 'accuracy' (also known as 'fidelity').  The following graphic, from this blog post, nicely demonstrate the difference between these two characteristics.

Other measurement characteristics include stability (repeatability from measurement to measurement), resolution (number of significant digits), sensitivity (ability to detect very small signals), linearity, range (from smallest valid value to largest valid value), and sampling rate (time slice or number of samples to establish a valid measurement).

Quick links to "Ten Dimensions" resources for #RSAC folks

This post is aimed at folks attending my RSA Conference talk on Wednesday, but could be useful for anyone who wants to catch up on the topics.

My talk is at 10:40am - 11:00am in Moscone West, Room: 2020.  Immediately after the talk, I'll be moving to the "Continuing the Conversation" space in the 2nd floor lobby of Moscone West.  I'll be wearing a black EFF hat, in case you want to pick me out of a crowd.

This is 20 minute talk, so it will only be an introduction to the topics.  My main goal is to stimulate your interest to learn more and to dig into these resources:

• RSAC slides: "Ten Dimensions of Security Performance for Agility and Rapid Learning"
• New blog post: "How to aggregate metrics into a performance index or score" (Thomas Scoring System, TSS) -- detailed tutorial
• New demo Excel spreadsheet for TSS, explained in the previous blog post
• Intro page to the Ten Dimensions, with links to all detail pages: "Ten Dimensions of Cyber Security Performance"

Not directly related to the above, but here's the slides for the talk I gave Monday at BSides-SF:

• "Get a Grip on Unintended Consequences" -- best in PPTX so you can read the stories in the speaker notes.  Here's an audio recording, starting at 4:17:24.  (FYI, Adam Shostack gave a nice talk in the hour before me on Threat Modeling.)

If we don't connect at the conference for some reason, feel free to email me at russell ♁ thomas ❂ meritology ♁ com.  (Earth = dot; Sun = at)

And if you've come this far and you aren't following me on twitter -- @MrMeritology -- what's wrong with you?  Follow, already! ☺

How to aggregate ground-truth metrics into a performance index

My remix of a painting by William Blake,
with the Meritology logo added. Get it?
He's shedding light on an impossible shape.
(Click to enlarge)

The general problem is this:

How can we measure aggregate performance on an interval or ratio scale index when we have a hodge-podge of ground-truth metrics with varying precision, relevance, reliability, and that are incommensurate with each other?

Here's a specific example from the Ten Dimensions:

How can we measure overall Quality of Protection & Controls if our ground-truth metrics include false positives percentages, false negatives percentages, exceptions number of exceptions, various "high-medium-low" ratings, audit results, coverage percentages, and a bunch more?

I've been wrestling with this problem for a long time, both in information security and elsewhere.  So have a lot of other people.  I while back I had an insight that the solution may be to treat it as an inference problem, not a calculation problem (described in this post). But I didn't work out the method at that time.  Now I have.

In this blog post, I'm introducing a new method.  At least I think it's new because, after much searching, I haven't been able to find any previously published papers. (If you know of any, please contact me or comment to this post.)

The new method is innovative but I don't think it's much more complicated or mathematically sophisticated than the usual methods (weighted average, etc.), but it does take a change in how you think about metrics, evidence, and aggregate performance.  Even though all the examples below are related to information security, the method is completely general.  It can apply to IT, manufacturing, marketing, R&D, governments, non-profits... any organization setting where you need to estimate aggregate performance from a collection of disparate ground-truth metrics.

This post is a tutorial and is as non-technical as I can make it. As such, its is on the long side, but I hope you find it useful.  A later post will take up the technicalities and theoretical issues. (See here for Creative Commons licensing terms.)

The Cost of a Near-miss Data Breach

[This post originally appeared in the New School of Information Security blog on October 6, 2009.  This idea was incorporated in the breach impact model presented in this paper, where you can find more details: ]

How Bad Is It?–A Branching Activity Model to Estimate the Impact of Information Security Breaches

If one of your security metrics is Data Breach Cost, what is the cost of a near miss incident? This seemingly simple question gets at the heart of security metrics problem.

Jerry escapes death, but is it cost-free?

Consider the gleeful Jerry Mouse in this cartoon. Tom the Cat has just missed in his attempt to swat Jerry and turn him into mouse meat. Is there any cost to Jerry for this near miss? Is Jerry’s cost any different than if he was running with Tom no where in sight?

By “near miss” I mean a security incident or sequence of incidents that could have resulted in a severe data breach (think TJX or Heartland), but somehow didn’t succeed. Let’s call the specific near-miss event “NM” for short. For sake of argument, let’s assume that the lack of attack success was due to dumb luck or attacker mistakes, not due to brilliant defenses or detection. Let’s say that you only discover NM long after the events took place. For simplicity let’s assume that discovering NM doesn’t result in any extraordinary costs, meaning that out-of-pocket costs are the same just before and immediately after NM. Finally, assume that your expected cost of a successful large-scale data breach is on the order of tens of millions, with the worst case being hundreds of millions of dollars.

How much does NM cost? The realist answer is “zero”. (Most engineers are realists, by disposition and training.) There is a saying in street basketball that expresses the realist philosophy about losses and associated costs: “No blood, no foul”. If you ask your accountants to pour over the spending and budget reports, they will probably agree. Case closed, right?

Not so fast….

Tutorial: How to Value Digital Assets (Web Sites, etc.)

[This originally appeared in the New School of Information Security Blog in two posts, Oct 20 and 23, 2009]

Many security management methods don’t rely on valuing digital assets.  They get by with crude classifications (e.g. “critical”, “important”, etc.).  Moreover, I dont believe that it’s absolutely necessary to calculate digital asset values to do quantitative risk analysis.  But if you need to do financial justification or economic analysis of security investments or alternative architectures then you might need something more precise and defensible.

This tutorial article presents one method aimed at helping line-of-business managers (“business owners” of digital assets) make economically rational decisions. It’s somewhat simplistic, but it does take some time and effort. Yet it should be feasable for most organizations if you really care about getting good answers.

Warning: No simple spreadsheet formulas will do the job. Resist the temptation to put together magic valuation formulas based on traffic, unique visits, etc.

(This is a long post, so read on if you want the full explanation…)

On the performance value of "cyber hygiene"

One idea that keeps coming up in the NIST Cyber Security Framework process is that we should be collectively promoting good "cyber hygiene" -- common things that everyone should be doing by habit to protect themselves on-line.  Analogies are made to personal health and hygiene and also personal safety (auto seat belts).  Vinton Cerf claims to have coined the term.  It is being widely promoted in cyber security awareness programs, including by outgoing DHS Secretary Janet Napolitano at this public event.  There are non-profit organizations focused on it, e.g. Stay Safe Online and Stop, Think, Connect. There's even a certificate in cyber hygiene offered.  These are often oriented at consumers and individuals, but the same ideas are being promoted for organizations, including those in critical infrastructure industries.

A real "cyber hygiene" promotion poster.  Let's all be smart chipmunks!

While most people seem to believe that it is possible to define "good cyber hygiene" and also worthwhile to promote it, not everyone agrees.  One commentator believes it puts too much burden on individuals and distracts us from the institutional and systematic forces that create or perpetuate the risks in the first place.

Of course, I have to try to answer these questions: where does "cyber hygiene" fit in to the proposed Ten Dimensions of Cyber Security Performance?  Can we define "good hygiene practices" in each dimension that serve as the common baseline for all organizations, as a minimum acceptable performance level, as a common entry level at the lowest level of maturity, or similar?

In my opinion, it is possible to define a common set of "cyber hygiene" practices for most individuals and most organizations.  They are good.  Do them.  But don't think you are achieving adequate or even minimum acceptable cyber security performance in an organization by simply implementing good "cyber hygiene".  At best, "cyber hygiene" is a set of practices that helps your organization be "anti-stupid".

Nice example of "Execution & Operations"

Ally Miller wrote a nice post called "Quant Ops" describing the relationship between DevOps and risk, which fits nicely into Dimension 5. Effective/Efficient Execution & Operations.  Best of all, she works through an example of how it works in practice.  Give it a read.

The key to measuring cyber security: inferences, not calculations

(It's getting very late here, so my apologies for any slips or gaps in communication.)

In many of the previous posts on the Ten Dimensions of Cyber Security Performance, I've hinted or suggested that these could be measured as a performance index.  But I'm sure many readers have frustrated because I haven't spelled out any details or given examples.  Still other readers will be skeptical that this can be done at all.

Sorry about that.  There's only so much I can put in each post without them becoming book chapters. In this post, I'll describe an approach and general method that I believe will work for all or nearly all the performance dimensions.  At the time of writing, this is in the idea stage, and thus it needs to be tested and proven in practice.  Still, I suggest it's worthy of consideration.

[Update 7/19/2013:  After some searching in Google Scholar, I discovered that the method I'm suggesting below is called Bayesian Structural Equation Modeling.  I'm very glad to see that it is an established method that has both substantial theory and software tool support.  I hope to start exploring it in the near future.  I'll post my results as I go.]

Here are my ideas on how to measure performance in each dimension.

Ten Dimensions of Cyber Security Performance

I am proposing a framework for managing cyber security performance.  As a preview, here are the ten dimensions:

1. Optimize Exposure: attack surface and vulnerabilities, including assets, people, processes, & technologies
2. Effective Threat Intelligence: understanding the threat agents 
3. Effective Design & Development: security & privacy by design 
4. Quality of Protection & Controls
5. Effective/Efficient Execution & Operations 
6. Effective Response, Recovery, & Resilience
7. Effective External Engagement: responsibilities and risk drivers
8. Effective Learning & Agility: OODA at an organization level
9. Optimize Total Cost of Risk: (loss distribution approach)
10. Responsibility & Accountability: including governance and compliance

Each of the ten dimensions are explored in subsequent posts (see links above).  The interactions among the first six dimensions are discussed in a post called "Operational Cyber Security & Single Loop Learning".  The interactions among the second four dimensions are discussed in "Agile Cyber Security and Double Loop Learning".

Here is are the slides of the diagram, built dimension by dimension.

These posts might be especially interesting to folks who engaged in any of the Cyber Security Framework processes now underway in the US (NIST), EU, or the UK.

(Comment: I acknowledge that this framework is a bit complicated.  My friend Jack Whitsitt   (@sintixerr) has suggested that we need "crayon version" because anything more complicated will just confuse people.  I concur, but I may have to flesh out this complicated version before I can get to something as simple as Plan-Do-Check-Act.)"