Ideal book for self-study: "Doing Bayesian Data Analysis"

In this post, I'd like to heartily recommend a book for anyone doing self-study who doesn't have much statistics or math in their background:

• Kruschke, J. (2010). Doing Bayesian Data Analysis: A Tutorial Introduction with R and BUGS. Academic Press.

This book is head-and-shoulders better than the others I've seen.  I'm using it myself right now.  Here's what's good about it:

• It builds from very simple foundations.
• Math is minimized.  No proofs.
• From start to finish, everything is demonstrated through R programs. Anyone learning statistics today should be learning a statistics programming language at the same time.  R is the most popular choice and by some measures the best choice.
• It helps you learn Empirical Bayesian methods from every angle.  It does great both with the fundamental concepts and the practical applications.
• It takes you as far as you want to go, at least into advanced territory if you want.  But you don't have to read the whole textbook to benefit.

For what it's worth, this book was voted most popular introductory book on Stack Exchange.

Quick links to "Ten Dimensions" resources for #RSAC folks

This post is aimed at folks attending my RSA Conference talk on Wednesday, but could be useful for anyone who wants to catch up on the topics.

My talk is at 10:40am - 11:00am in Moscone West, Room: 2020.  Immediately after the talk, I'll be moving to the "Continuing the Conversation" space in the 2nd floor lobby of Moscone West.  I'll be wearing a black EFF hat, in case you want to pick me out of a crowd.

This is 20 minute talk, so it will only be an introduction to the topics.  My main goal is to stimulate your interest to learn more and to dig into these resources:

• RSAC slides: "Ten Dimensions of Security Performance for Agility and Rapid Learning"
• New blog post: "How to aggregate metrics into a performance index or score" (Thomas Scoring System, TSS) -- detailed tutorial
• New demo Excel spreadsheet for TSS, explained in the previous blog post
• Intro page to the Ten Dimensions, with links to all detail pages: "Ten Dimensions of Cyber Security Performance"

Not directly related to the above, but here's the slides for the talk I gave Monday at BSides-SF:

• "Get a Grip on Unintended Consequences" -- best in PPTX so you can read the stories in the speaker notes.  Here's an audio recording, starting at 4:17:24.  (FYI, Adam Shostack gave a nice talk in the hour before me on Threat Modeling.)

If we don't connect at the conference for some reason, feel free to email me at russell ♁ thomas ❂ meritology ♁ com.  (Earth = dot; Sun = at)

And if you've come this far and you aren't following me on twitter -- @MrMeritology -- what's wrong with you?  Follow, already! ☺

How to aggregate ground-truth metrics into a performance index

My remix of a painting by William Blake,
with the Meritology logo added. Get it?
He's shedding light on an impossible shape.
(Click to enlarge)

The general problem is this:

How can we measure aggregate performance on an interval or ratio scale index when we have a hodge-podge of ground-truth metrics with varying precision, relevance, reliability, and that are incommensurate with each other?

Here's a specific example from the Ten Dimensions:

How can we measure overall Quality of Protection & Controls if our ground-truth metrics include false positives percentages, false negatives percentages, exceptions number of exceptions, various "high-medium-low" ratings, audit results, coverage percentages, and a bunch more?

I've been wrestling with this problem for a long time, both in information security and elsewhere.  So have a lot of other people.  I while back I had an insight that the solution may be to treat it as an inference problem, not a calculation problem (described in this post). But I didn't work out the method at that time.  Now I have.

In this blog post, I'm introducing a new method.  At least I think it's new because, after much searching, I haven't been able to find any previously published papers. (If you know of any, please contact me or comment to this post.)

The new method is innovative but I don't think it's much more complicated or mathematically sophisticated than the usual methods (weighted average, etc.), but it does take a change in how you think about metrics, evidence, and aggregate performance.  Even though all the examples below are related to information security, the method is completely general.  It can apply to IT, manufacturing, marketing, R&D, governments, non-profits... any organization setting where you need to estimate aggregate performance from a collection of disparate ground-truth metrics.

This post is a tutorial and is as non-technical as I can make it. As such, its is on the long side, but I hope you find it useful.  A later post will take up the technicalities and theoretical issues. (See here for Creative Commons licensing terms.)

Two new #InfoSec books that could transform your way of thinking

Happiness is having great colleagues and collaborators.  I'm very happy to recommend to you two new books by three of my favorite colleagues -- Jay Jacobs (@jayjacobs), Bob Rudis (@hrbrmstr), and Adam Shostack (@adamshostack).  These books not only do a great job covering the topics, they could also transform your way of thinking.

• Data-driven Security -- Analysis, Visualization, and Dashboards
• Threat Modeling: Designing for Security

preso: Big 'R' Risk Management - from concept to pilot implementation

Here's the presentation (pdf) that I'm giving Monday at SIRAcon in Seattle.  This extends the ideas presented in the post "Risk Management: Out with the Old, In with the New!". This presentation presents some specifics on how to get started implementing the Big 'R' approach. It's even got a illustrative case toward the end featuring patch management and exceptions, shown in this figure (click to enlarge)

Example of Causal Dynamic Analysis, in this case Patch Management & Exceptions
(click to enlarge)

Tutorial: How to Value Digital Assets (Web Sites, etc.)

[This originally appeared in the New School of Information Security Blog in two posts, Oct 20 and 23, 2009]

Many security management methods don’t rely on valuing digital assets.  They get by with crude classifications (e.g. “critical”, “important”, etc.).  Moreover, I dont believe that it’s absolutely necessary to calculate digital asset values to do quantitative risk analysis.  But if you need to do financial justification or economic analysis of security investments or alternative architectures then you might need something more precise and defensible.

This tutorial article presents one method aimed at helping line-of-business managers (“business owners” of digital assets) make economically rational decisions. It’s somewhat simplistic, but it does take some time and effort. Yet it should be feasable for most organizations if you really care about getting good answers.

Warning: No simple spreadsheet formulas will do the job. Resist the temptation to put together magic valuation formulas based on traffic, unique visits, etc.

(This is a long post, so read on if you want the full explanation…)