If you have a spare 1 hr 40 min. *, you might want to listen my interview on the "Loopcast" podcast (below). The host is Sina Kashefipour (@rejectionking on Twitter). * Personally, I think I sound better at 1.5X speed, but then again I listen to most podcasts at 1.5X speed.
In this podcast, I reference the following websites and resources:
Like Curly from the "Three Stooges" said, "I'm a victim of circumstances!"
There is a type of "...Swan" that is not surprising or extreme in its aggregate effect, but is extremely surprising to a particular entity that was considered to be outside the scope of the main process. A bomb is supposed to kill enemy soldiers, not your own soldiers.
Summary: It
doesn't make sense to label any set of events as "Black Swans". It's
not the events themselves, but instead they are processes that involve
generating mechanisms, our evidence about them, and our method of
reasoning that make them unexpected and surprising.
Definition
A "Splattered Swan" is a process where:
The generating process involves a very powerful force (i.e. penal, constraining, or damaging force) with less than perfect aim.
The evidence
are official rules, specifications, or scope, or experience that is limited what is intended;
The method of reasoning
are based on the assumption that the aim will be perfect and error-free, or that errors will be "well behaved".
Main Features
A Splattered Swan arises when a very powerful system is prone to misfiring in very bad ways, causing damage to some entities that are considered "safe" or "out of bounds" by normal reasoning. That these outcomes are extreme or surprising are basically due to failures to understand the total system and the ways it can fail.
Two key features of Splattered Swans are 1) critical error conditions are excluded from reasoning on principle and 2) those errors are potentially severe, even the first time. A lot of systems adapt by trial and error, but that only works of the magnitude of errors (i.e. aim) is relatively small and the magnitude of collateral damage is also relatively small. Consider airplane bombers from World War II aiming to kill enemy troops that are located near allied troops. Even though they had bomb sights, the bombers were notoriously inaccurate. With ordinary large bombs, the risk of "friendly fire" (i.e. killing your own troops) is high. If the bomber is carrying a single atomic bomb, then the risk of "friendly fire" becomes extremely high, because you only get one chance to aim and drop and there is no feedback from previous attempts. Plus the damage process is extreme. In the other direction, if there are several bombers, and the first bombers drop flairs instead of bombs, then the cost of error is small and the opportunity of corrective feedback has the potential to dramatically reduce the risk of "friendly fire".
Another important feature of Splattered Swans are the blind spots created by the "official" or "intended" definition of the system of interest. This can lead analysts and decision-makers to never even consider the possibility of collateral damage or unintended consequences.
One Example
"Offensive cyber" , a.k.a. "hack back" is an example from the domain of cyber security. There are many flavors of offensive cyber, but the most extreme involve damaging the targets, either physically or digitally or both. Such extreme attacks might also be considered acts of war, a.k.a. "cyber war". Putting aside the ethics or advisability of offensive cyber, there is immense potential for collateral damage. First, it might be hard or impossible to attribute a given attack to the "real" threat agents or groups (a.k.a. "Black Hat"). They might operate through affiliates, mask or disguise their tools and infrastructure, and might even intentionally implicate a different agent or group in the "Indicators of Compromise" and other forensic evidence. Even if you can correctly identify the attacking group, it may be hard to attack them in a way that doesn't also do harm to socially-important entities or resources (e.g. cloud computing resources, networks, etc.). Finally, in corner-case situations there is also a non-zero potential for self-harm, where an offensive cyber attack backfires on the "White Hat" attacker.
From a planning and on-going management viewpoint, it is much harder to anticipate and control the side-effects of cyber attacks than it is for physical attacks.
How to Cope with Splattered Swan
It is relatively simple to cope with Splattered Swan systems. Don't take the "official" or "intended" system as a strict definition of what behavior or outcomes are possible. Use Scenario Planning or "What If?" analysis to look outside the "official" or "intended" to identify potential for collateral damage. Then look for ways to introduce error-correcting feedback or damage mitigations for the collateral damage. Another good mitigation is to reduce the intensity of the damage/punishment process.
In complex systems with many interacting elements or agents, most of the time the actions of individual agents "average out" and remain local. In some systems and in some circumstances, surprising aggregate or emergent behavior patterns arise.
Emergence is a common characteristic of such systems. But that alone doesn't qualify them as a "Swan". It requires several other factors that could, under the right circumstances, yield very surprising or cataclysmic outcomes. I call this the "Swarm-as Swan", since swarming behavior (birds, fish, insects) is one well-known type of emergent phenomena, but this category is explicitlynot limited to swarm phenomena.
Summary: It
doesn't make sense to label any set of events as "Black Swans". It's
not the events themselves, but instead they are processes that involve
generating mechanisms, our evidence about them, and our method of
reasoning that make them unexpected and surprising.
Definition
A "Swarm-as-Swan" is a process where:
The generating process involves a large-scale Complex Adaptive System that has regions in the state space where collective and/or emergent phenomena become dominant, leading to collective behavior that is dramatically different from the behavior in the "normal" regions of state space.
The evidence
are patterns of system behavior and interaction at various scales (individual, group, collective) and especially surprisingly different patterns, including downward causation and varieties of self-organization and information processing;
The method of reasoning are mental models of the system, whether formal or informal, sophisticated or common sense, and the implications of those models on what behaviors are "normal" and expected vs. what is surprising.
Main Features
The field of Complexity Science has grown and blossomed over the last 30 years. This Wikipedia article gives a good summary, along with the central theme of Complex Adaptive Systems (CAS). From that article, the common characteristics of CAS:
The number of elements is sufficiently large that conventional descriptions (e.g. a system of differential equations) are not only impractical, but cease to assist in understanding the system. Moreover, the elements interact dynamically, and the interactions can be physical or involve the exchange of information
Such interactions are rich, i.e. any element or sub-system in the system is affected by and affects several other elements or sub-systems
The interactions are non-linear: small changes in inputs, physical interactions or stimuli can cause large effects or very significant changes in outputs
Interactions are primarily but not exclusively with immediate neighbors and the nature of the influence is modulated
Any interaction can feed back onto itself directly or after a number of intervening stages. Such feedback can vary in quality. This is known as recurrency
The overall behavior of the system of elements is not predicted by the behavior of the individual elements
Such systems may be open and it may be difficult or impossible to define system boundaries
Complex systems operate under far from equilibrium conditions. There has to be a constant flow of energy to maintain the organization of the system
Complex systems have a history. They evolve and their past is co-responsible for their present behavior
Elements in the system may be ignorant of the behavior of the system as a whole, responding only to the information or physical stimuli available to them locally
While all CAS are, in principle, capable of emergent behavior, not all are capable of big surprises in behavior that we require for our "...Swans" series. Roughly, there are three levels of emergent phenomena
Level 1: Emergent behavior -- behavior of large numbers of individuals becomes interdependent and mutually influencing, far beyond the range of causal and information interaction, including some downward causation where the collective shapes the individuals. Examples: flocks of birds, schools of fish.
Level 2: Emergent functional structures -- the formation of stable networks of individuals that constitute functional subsystems. "Functional" means they do some work beyond just collective behavior of Level 1. An excellent example is the "glider" phenomena in Conway's Game of Life (a type of cellular automata).
A single Gosper's glider gun creating "gliders"
Level 3: Emergence against a model -- similar to Level 2, but the "stable functional subsystems" have information processing and self-sustaining capabilities (including possibly metabolizing energy and repairing/regenerating structures). In a real sense, Level 3 systems "take on a life of their own", at least for an extended period. Example: emergent subsystems that function as regulators (e.g. thermostat), communication systems (e.g. encoding, decoding, transmission), pattern matching, optimization, etc. The human immune system has some of these capabilities.
As you move up these levels, the nature of emergent phenomena changes dramatically, from mildly surprising and interesting at Level 1 up to "HOLY COW" at Level 3. But our understanding of CAS is most at Level 1, some at Level 2, and only a little at Level 3. Put another way, if you create a hundred or thousand different CAS in a computational laboratory, most would only exhibit Level 1 emergent phenomena, a few would exhibit Level 2 emergent behavior, and only a very few, under narrow circumstances, would have Level 3 capabilities. (In his book A New Kind of Science, Stephen Wolfram did exactly this investigation for all simple cellular automata.) Like all of the "...Swans", it's just as important to understand the evidence that we use to understand these systems (i.e. CAS), and also our methods of reasoning. It's the combination of all three that give rise to the surprising/shocking/extreme behavior that we associate with "...Swans". The most common evidence people pay attention two is either individual-based behaviors and interactions and the most common collective behavior patterns and distributions. If the states of the CAS are in "low complexity" regions of the state space (i.e. not in one of the three Levels, above), then people may not even recognize that the CAS is capable of complex emergent phenomena. The reverse is also true. If the CAS is normally in a highly coherent, highly functional state then people may not observe or understand the micro-level behavior that supports that phenomena. The evidence we need most is the location of "phase transitions" in state space, where the CAS shifts dramatically from one regime to another. Unfortunately for us mortal humans, it's almost impossible to know in advance where the important phase transitions are in CAS, especially the Level 2 and 3 CAS. Our methods of reasoning about CAS fall into three categories: 1) Intuitive (i.e patterns of "normal" behavior with small deviations,
naïve causal models, "folk wisdom", etc.); 2) Linear Models (i.e. the standard tools of science up to ~1990); 3) Non-linear Models, including Agent-based Modeling. Methods 1) and 2) are most common, and work well in "normal" circumstances, but are very prone to catastrophic failures of reasoning when the CAS enters a new, unfamiliar regime of emergent behavior. Method 3) is specifically designed to understand CAS in all their complexity, but they aren't a "magic bullet" that completely eliminate the potentials for surprise or extreme outcomes. One huge difference between Method 2) "Linear Models" and Method 3) "Non-linear Models" is that the that Method 3) usually does not yield a forecast or prediction of system behavior in the same way that Method 2) does. Instead it can help us understand when and why the CAS will change regimes, which is still very useful information to understand potential surprises.
Examples
In the previous section I mentioned some illustrative examples. But here I'll mention two "biggies".
The modern economy has been characterized and studied as a Complex Adaptive System (CAS), especially to understand innovation and crises, especially societal/economic collapse. Some of the first books published by the Santa Fe Institute in the late `80s were titled "The Economy as a Complex Adaptive System". Mass uprisings and mass revolutions are other classes of phenomena that benefit from study as CAS. I won't go into detail here, but if you are curious, you might read up on these cases:
The first step is to recognize that the system you are dealing with has the characteristics of a Complex Adaptive System. Start with the Wikipedia page, then read some of the general references listed at the bottom. This will give you the basic knowledge plus some exposure to many types of CAS. The second step is to characterize the types of emergent behavior and structures that are within the "possibility space" of the CAS. But stay away from "magical thinking" and "conspiracy theories". The third step is to apply modeling tools that are appropriate to the complexity of the CAS. Linear models are fine for what they do, but don't try to use them to identify "phase transitions" from simple to complex behavior, etc. If you aren't mathematically inclined or are not comfortable programming your own Agent-based Models (ABM), you can at least read books and papers that utilize these models and learn from the pros who built them and analyzed them. Even better, you can play with them yourself using the Model Library that comes with NetLogo (free and open source). Each model is controlled by sliders and buttons, and comes with documentation that guides you how to use it, how to interpret it, and how to explore it.
Here is a video demo of the NetLogo "Cyber Security Investment Game", as a work in progress. This run is a 6/3 multiplayer game, meaning 6 cooperative Defenders and 3 adversarial Attackers. Defenders play a 2-player game with everyone else, while Attackers only play 2-player games with Defenders. It's a complicated game, with up to 400 possible "moves" for each agent at each step, and also a dynamic game, meaning the structure of the game can change dynamically during the course of play. (Game Theory Purists will probably hate it for that reason!)
Here's a video of 1,700 time steps, with 20x speed-up.
What you see in this simulation is an ecosystem that provides a positive environment for both Defenders and Attackers. That is, they both have consistently positive payoffs (see bar graph, center left).
While the overall trends of payoffs are pretty clear (bottom graph), there is an "unruly" back and forth between attackers and defenders. Crucially, the time series of payoffs is non-stationary for all agents (see center graphs, third and fourth from the top). Simply, non-stationary means that the probability distribution for each payoff changes over time. You can see a sample distribution of payoffs in the two histograms, center right. You'll notice them changing shape (going from skew to symmetric) and also changes in mean and standard deviation ("SD"). Non-stationarity has implications for risk estimation, as I will detail in the up-coming WEIS paper.
In terms of progress toward the goal, I would say this is not yet a model of cyber security investment. It models competitive relationships rather than not host-parasite relationships which I believe are close to the true nature of cyber security ecosystems. The good news is that I believe I know what extensions and modifications need to be made. I'll save that for a later post.
Here are some key features of this version of the model:
Three levels of investment: 1) architecture/infrastructure; 2) capabilities; 3) practices/routines (a.k.a. "moves")
Asymmetric number and variety of "moves" for Defenders and Attackers
Parametric control over diversity of "moves" and investments by Defenders and Attackers
Yes, the model is getting complicated. But I hope this richness will be rewarded when crucial results are revealed in experiments. We shall see...
<update>
Here is the same run after 4,000 time steps. No sign of equilibrium or stationary time series.
Summary: Time and uncertainty are essential features of any model of the "game of cyber security". Models that do not include them as central features are not fit for purpose. But, yes, they do make life more difficult for modelers and their audiences. While I make the case that both are essential, I leave open the question as to what is the most parsimonious method or treatment.
How fast does the space of possibilities expand? This question is explored in the following paper (free download):
Tria, F., Loreto, V., Servedio, V. D. P., & Strogatz, S. H. (2014). The dynamics of correlated novelties. Nature Science Preport, 4. (http://dx.doi.org/10.1038/srep05890)
From the abstract:
Novelties are a familiar part of daily life. They are also fundamental to the evolution of biological systems, human society, and technology. By opening new possibilities, one novelty can pave the way for others in a process that Kauffman has called “expanding the adjacent possible”. The dynamics of correlated novelties, however, have yet to be quantified empirically or modeled mathematically. Here we propose a simple mathematical model that mimics the process of exploring a physical, biological, or conceptual space that enlarges whenever a novelty occurs. The model, a generalization of Polya's urn, predicts statistical laws for the rate at which novelties happen (Heaps' law) and for the probability distribution on the space explored (Zipf's law), as well as signatures of the process by which one novelty sets the stage for another.
I've written a NetLogo program to replicate their model, available here. The code for the model is quite simple. A majority of my code is for a "pretty layout", which is a schematic version of a "top-down view" of the urn. Here's a video of a single run
Full screen with controls. (click to enlarge)
The charts on the top and center right show the frequency distribution by ball type (a.k.a. "color"). These are log-log plots, so a straight line (declining) is signature of a power law distribution, while a gradually curving (concave) is signature of lognormal or similar distribution with somewhat thinner tail. Sharply declining curve is signature of a thin tailed distribution such as Gaussian.
So what?
This model will be useful in my dissertation because I need mechanisms to endogenously add novelty -- i.e. expand the possibility space based on the actions of agents in the simulated world, and not simply as external "shocks".
This is essential for modeling cyber security because some people claim that quantitative risk management is impossible in principle because of intelligent adversaries who can generate and exploit novel strategies and capabilities.
Say you are an entrepreneurial sort of person who wants to really change the world of cyber security. Problem: nobody seems to know where the game-changing innovation is going to come from. Is it technology? Is it economics? Is it law and policy? Is it sociology? Maybe combination, but what? And in what sequence?
If you aim for institutional innovation, then at some point you are going to need to take sides in the great "Quant vs. Non-quant" debate:
Can cyber security and risk be quantified?
If "yes", how can quantitative information be used to realize security to significantly improve outcomes?
Whether you choose Quant or Non-quant, you will need some tools and methods to advance the state of the art. But how do you know if you are choosing the right tools, and using them well? (Think about the difference between Numerology and Calculus as they might be applied to physics of motion.)
Whoever makes sufficient progress toward workable solutions will "win", in the sense of getting wide-spread adoption, even if the other is "better" in some objective sense (i.e. "in the long run").
I examine this innovation race in a book chapter (draft). The book will probably come out in 2016.
Abstract:
"The focus of this chapter is on how the thoughts and actions of actors coevolve when they are actively engaged in institutional innovation. Specifically: How do innovators take meaningful action when they are relatively ‘blind’ regarding most feasible or desirable paths of innovation? Our thesis is that innovators use knowledge artifacts – e.g. dictionaries, taxonomies, conceptual frameworks, formal procedures, digital information systems, tools, instruments, etc. – as cognitive and social scaffolding to support iterative refinement and development of partially developed ideas. We will use the case of institutional innovation in cyber security as a way to explore these questions in some detail, including a computational model of innovation."
Your feedback, comments, and questions would be most welcome.
The computational model used is called "Percolation Models of Innovation". Here is the NetLogo code of the model used in the book chapter. Below are some figures from the book chapter.
Innovation as percolation. Progress moves from bottom to top. Each column is a "technology",
and neighboring columns are closely related. This version (S&V 2005) only models
rate of progress and distribution of "sizes", not anything about the technology or
trajectory of innovation.
A screen shot of the user interface. Three different models can be selected (upper left).
I have written a NetLogo version of the random game model of Galla & Farmer (2012) (free download). It has been uploaded to the NetLogo community library and should appear in a day or so. Read on if you are interested in Game Theory, esp. learning models and computational methods.
Chaotic dynamics in a complicated game. The payoffs are negatively correlated (-0.7) and memory for learning is long (alpha ≈ 0). Notice the squiggly lines in the time series plots (lower right). Each line is probability for a given move.
If the game were in equilibrium, these lines would be flat. (click for larger image)
Here are the slides I'm presenting today at B-Sides San Francisco (4pm). I suggest that you download it as PPTX because it is best viewed in PowerPoint so you can read the stories in the speaker notes.
This is a wonderful 30 minute lecture that should be interesting to anyone in information security, risk management, operations, and especially CIOs and CISOs. He gives very good explanations about why agility and learning are so important to resilience.
