SIRAcon: "Probabilistic models of breach impact – combining theory and empirical data"

Here are my slides from my SIRAcon talk.

Here's a slide that got a lot of attention.  (Humorous, of course)

B-Sides SF Talk

Here is the demo spreadsheet I'll be using in today's B-Sides SF talk on the Thomas Scoring System (TSS):

• InfoSec Maturity Assessment (TSS) V1.xlsx

Download the spreadsheet and open in Microsoft Excel 2008 or later.  It uses conditional formatting and cell data validation, but no macros or other advanced features.  The sheets are protected to avoid data entry errors, but there is no password.

This is a realistic, fully functional implementation of the TSS applied to a general case: scoring the maturity of a company's information security capability.

Boomer weasel words: 'high net worth individuals' as euphemism for 'rich people'

(Source)

For some time, I have been noticing that rich people and the people who sell services to them don't use the phrase "rich people" any more.  Instead, they say "high net worth individuals".  When did this happen and why?

(You might compare this to my previous post regarding "Baby on Board" signs.)

The following graphs are from Google Ngram Viewer, which show relative frequency of word phrases in American English books up to the year 2008. Notice that the phrase "high net worth individuals" appears first around 1980.

(click to enlarge)

Ideal book for self-study: "Doing Bayesian Data Analysis"

In this post, I'd like to heartily recommend a book for anyone doing self-study who doesn't have much statistics or math in their background:

• Kruschke, J. (2010). Doing Bayesian Data Analysis: A Tutorial Introduction with R and BUGS. Academic Press.

This book is head-and-shoulders better than the others I've seen.  I'm using it myself right now.  Here's what's good about it:

• It builds from very simple foundations.
• Math is minimized.  No proofs.
• From start to finish, everything is demonstrated through R programs. Anyone learning statistics today should be learning a statistics programming language at the same time.  R is the most popular choice and by some measures the best choice.
• It helps you learn Empirical Bayesian methods from every angle.  It does great both with the fundamental concepts and the practical applications.
• It takes you as far as you want to go, at least into advanced territory if you want.  But you don't have to read the whole textbook to benefit.

For what it's worth, this book was voted most popular introductory book on Stack Exchange.

Mining only 'digital exhaust', Big Data 1.0 won't revolutionize information security

I was asked during this interview whether 'Big Data' was revolutionizing information security.  My answer was, essentially, 'No, not yet'. But I don't think I did such a great job explaining why and where the revolution will come from, if it comes.

Basically, Big Data 1.0 in information security is today focused on mining 'digital exhaust' -- all the transactional data emitted and logged by computing, communications, and security devices and services.  (The term "data exhaust" was probably coined in 2007 by consultant Jerry Michalski, according to this Economist article.)  This can certainly be useful for many purposes but I don't think it is or will be revolutionary.  It will help tune spam filters, phishing filters, intrusion detection/prevention systems, and so on, but it won't change anything fundamental about how firms architect security, how they design and implement policies, and it does almost nothing on the social or economic factors.

Here's a great essay that explains why Big Data 1.0 isn't revolutionary, and what it will take to make it revolutionary.  Though it's not about information security, it doesn't take much to extend his analysis to the InfoSec domain.

Huberty, M. (2014). I expected a Model T, but instead I got a loom: Awaiting the second big data revolution. Prepared for the BRIE-ETLA Conference, September 6-7, 2013, Claremont California.

Huberty points toward Big Data 2.0 which could be revolutionary:

"...we envision the possibility of a [Big Data 2.0]. Today, we can see glimmers of that possibility in IBM’s Watson, Google’s self-driving car, Nest’s adaptive thermostats, and other technologies deeply embedded in, and reliant on, data generated from and around real-world phenomena. None rely on “digital exhaust”. They do not create value by parsing customer data or optimizing ad click-through rates (though presumably they could). They are not the product of a relatively few, straightforward (if ultimately quite useful) insights. Instead, IBM, Google, and Nest have dedicated substantial resources to studying natural language processing, large-scale machine learning, knowledge extraction, and other problems. The resulting products represent an industrial synthesis of a series of complex innovations, linking machine intelligence, real-time sensing, and industrial design. These products are thus much closer to what big data’s proponents have promised–but their methods are a world away from the easy hype about mass-manufactured insights from the free raw material of digital exhaust.

[...]

The big gains from big data will require a transformation of organizational, technological, and economic operations on par with that of the second industrial revolution. " [emphasis added]

---

Highlighting somewhat different themes in the context of Digital Humanities, Brian Croxall presents an insightful blog post called "Red Herrings of Big Data", which includes slides and this 2 minute video:



Here are his three 'red herrings' (i.e. distractions from the most promising trail), turned around to be heuristics:

Main message

Don't be naïve about Big Data in information security. To drive a revolution, it will need to be part of a much more comprehensive transformation of what data we gather in the first place and how data analysis and inference can drive results.  Just mining huge volumes of 'digital exhaust' won't do it.

How to aggregate ground-truth metrics into a performance index

My remix of a painting by William Blake,
with the Meritology logo added. Get it?
He's shedding light on an impossible shape.
(Click to enlarge)

The general problem is this:

How can we measure aggregate performance on an interval or ratio scale index when we have a hodge-podge of ground-truth metrics with varying precision, relevance, reliability, and that are incommensurate with each other?

Here's a specific example from the Ten Dimensions:

How can we measure overall Quality of Protection & Controls if our ground-truth metrics include false positives percentages, false negatives percentages, exceptions number of exceptions, various "high-medium-low" ratings, audit results, coverage percentages, and a bunch more?

I've been wrestling with this problem for a long time, both in information security and elsewhere.  So have a lot of other people.  I while back I had an insight that the solution may be to treat it as an inference problem, not a calculation problem (described in this post). But I didn't work out the method at that time.  Now I have.

In this blog post, I'm introducing a new method.  At least I think it's new because, after much searching, I haven't been able to find any previously published papers. (If you know of any, please contact me or comment to this post.)

The new method is innovative but I don't think it's much more complicated or mathematically sophisticated than the usual methods (weighted average, etc.), but it does take a change in how you think about metrics, evidence, and aggregate performance.  Even though all the examples below are related to information security, the method is completely general.  It can apply to IT, manufacturing, marketing, R&D, governments, non-profits... any organization setting where you need to estimate aggregate performance from a collection of disparate ground-truth metrics.

This post is a tutorial and is as non-technical as I can make it. As such, its is on the long side, but I hope you find it useful.  A later post will take up the technicalities and theoretical issues. (See here for Creative Commons licensing terms.)

Does a model and its data ever speak for themselves? No -- A reply to Turchin

This post is the first of a series to reply to Dr. Peter Turchin regarding his PNAS article (full text PDF -- free, thanks to Turchin & team), my letter to PNAS, and his PNAS letter reply.  I wrote a blog post here because I didn't think that Dr. Turchin's reply addressed my questions due to misunderstanding and I invited Dr. Turchin to engage in a colloquy via blog posts. I'm happy to say that Dr. Turchin wrote three blog posts (here, here, and here) in reply to my post, and this is my first reply.

While this post talks about interpreting simulation results, the general topic of data interpretation applies to all empirical research, and even data analysis in industry.  

What analysis do we really need to guide vulnerability management?

This is the first of a series of posts on the topic of doing quantitative risk analysis in the face of intelligent and adaptive adversaries.  Later posts will dig into research topics like combining risk analysis with game theory, but this first post is mostly a reaction to what other people have said recently.

Rafał Łoś recently posted an article, and then followed with a guest post from Heath Nieddu, with this general theme (paraphrasing and condensing):

It's senseless and distracting to attempt to use quantitative risk analysis to make decisions about vulnerability remediation, and even for information security as a whole.  Uncertainties about the future are too great; adversaries too agile and intelligent; and the whole quant risk endeavor is too complicated.  Keep it simple and stick with what you know for sure, especially the basics.

In this post I'm going to address some of the issues and questions this skeptical view raises, but I won't attempt a point-by-point counter argument.  For the record, there are many points I disagree with, plus many ideas that I think are confused or just mis-stated.  But I think the discussion will be best served by keeping focused on the main issues.

I'm also appearing on Rafał's podcast, Down the Rabbit Hole, along with some other SIRA members.  I'll let you know when it is posted for listening.

Baby Boomer on Board! -- a data-based exploration

Kindergarten graduations.  Play dates. First names like Zaiden (boy) and Bristol (girl). Driving kids to and from school in SUVs and minivans when they live in walking distance (e.g. 2 blocks).

And those "Baby on Board!" signs.

What "Baby on Board!" really signifies.

How did we get here?

This post is a data-based exploration into the origins of these cultural patterns in the White middle and upper-middle classes in the US.

(warning: long post, but with many charts and pictures)

The Bayesian vs Frequentist Debate And Beyond: Empirical Bayes as an Emerging Alliance

This is one of the best articles I've ever seen on the Bayesian vs Frequentist Debate in probability and statistics, including a description of recent developments such as the Bootstrap, a computationally intensive inference process that combines Bayesian and frequentist methods.

Efron, B. (2013). A 250-year argument: Belief, behavior, and the bootstrap. Bulletin of the American Mathematical Society, 50(1): 129-146.

Many disagreements about risk analysis are rooted in differences in philosophy about the nature of probability and associated statistical analysis.  Mostly, the differences center on how to handle sparse prior information, and especially the absence of prior information. "The Bayesian/frequentist controversy centers on the use of Bayes rule in the absence of genuine prior experience."

What's great about this article is that it presents the issue and alternative approaches in a simple, direct way, including very illuminating historical context.  It also presents a very lucid description of the advantages and limitations of the two philosophies and methods.

Finally, it discusses recent developments in the arena of 'empirical Bayes' that combines the best of both methods to address inference problems in the context of Big Data.  In other words, because of Big Data and the associated problems people are trying to solve now, pragmatics matter more than philosophical correctness.  Another example of empirical Bayes is Bayesian Structural Equation Modeling that I referenced in this post.

Where are the NLP or Text Mining Tools for Automated Frame Analysis?

I'd like to do Frame Analysis (Goffman 1976, Johnston 1995) on a medium-sized corpus of text (articles, speeches, blog posts and comments) and I'm looking for NLP or text mining tools to help with it.  Strangely, I can't find anything. All the examples of frame analysis in published research (e.g. Nisbet 2009) use purely manual methods or computer-augmented manual analysis.

Frame Analysis requires sophisticated semantic analysis, filtering, situational understanding, and inference on missing text. Near as I can tell, this level of sophistication is beyond the grasp of the common NLP and text mining tools.  Is this true?  If not, do any of you know of fully automated tools for Frame Analysis?

I should add that I have two use cases. First, the most demanding, is automatic identification of frames followed by text classification.  Second, more feasible, is automatic classification of texts given frame definitions and sample texts.  The latter fits the classic machine learning model of supervised learning, so I assume that as long as my training set is large enough and representative enough, I can probably find an adequate ML classification algorithm.

[Edit 7/23/2013:  This is the best summary I could find of available tools: Frame Analysis: Software]

__________

Goffman, E. (1976). Frame Analysis: An Essay on the Organization of Experience. Harvard University Press.

Johnston, Hank (1995). A methodology for frame analysis: from discourse to cognitive schemata. In Social Movements and Culture (pp. 217–246). University of Minnesota Press.

Nisbet, M. C. (2009). Communicating Climate Change: Why Frames Matter for Public Engagement, Environment. Science and Policy for Sustainable Development, 51(2): 12–23.

Nice example of "Execution & Operations"

Ally Miller wrote a nice post called "Quant Ops" describing the relationship between DevOps and risk, which fits nicely into Dimension 5. Effective/Efficient Execution & Operations.  Best of all, she works through an example of how it works in practice.  Give it a read.

Q: What marks the "Heartland"? A: Attitudes toward pantyhose substitutes

NC State has a very interesting interactive site called Dialect Survey Maps, based on data from the 122-question survey conducted by Bert Vaux, Department of Linguistics, University of Cambridge. The web interface was coded with Shiny and deployed using the hosting service provided by RStudio.

Maps for most questions show regional differences in terminology and idioms -- e.g. in most of the US, "tennis shoes" is generic name for soft/athletic shoes, except in New England where the term "sneakers" is preferred.

But the map that caught my eye was for this statement:

• 56. "Pantyhose are so expensive anymore that I just try to get a good suntan and forget about it."

Here's the map.  What I like is that agreeing with this statement is marks "The Heartland"  (excluding the Old South), a.k.a. "North-Middle America".  (Color code:  Blue is "acceptable" and red is "unacceptable".  Click to see larger version in a new window.)

It's interesting to see how this separates Iowa from Minnesota, Wisconsin from Illinois, North Dakota from South Dakota, and New York from Pennsylvania.