Grey Swans: Cascades in Large Networks and Highly Optimized/Critically Balanced Systems

A Grey Swan -- almost Black, but not quite. More narrowly defined.

This is the first of the series "Many Shades of Black Swans", following on the introductory post "Think You Know Black Swans? Think Again."

I'll define and describe each one, and maybe give some examples. Most important, each of these Shades will be defined by a mostly-unique set of 1) generating process(es); 2) evidence and beliefs; and 3) methods of reasoning and understanding.  As described in the introductory post, it's only in the interaction of these three that Black Swan phenomena arise. Each post will close with section called "How To Cope..." that, hopefully, will make it clear why this Many Shades approach is better than the all-lumped together Black Swan category.

This first one is named "Grey" because it's closest to Taleb's original concept before it got hopelessly expanded and confused.

Definition

 A "Grey Swan" as a stochastic process and estimation system where:

1. The generating process is effectively a stationary heavy-tailed distribution of magnitudes; 
2. The evidence is a comparatively small sample of history, with the taken-for-granted belief that the distribution is thin-tailed (i.e. not heavy-tailed);
3. The method of reasoning is frequentist statistics, using only available history and maybe also statistical methods that assume thin-tailed distributions.

A "stochastic process" is just a random draw from a distribution that repeats over time, resulting in time series of outcomes.  The outcomes are distributed according to the random process that generating process.  For example, if we repeatedly flip a coin and record the outcome -- "H" or "T" -- each time, resulting in a random time series like this:

"H T T H T T H T T T T H H H T H H T T H H T...".  

Likewise, the output of a stochastic process could be a series of magnitudes (numbers), a series of objects, or a series of patterns, each drawn from a probability distribution of outcomes.

Main Features

First, it's important to note that what is being generated is a magnitude, not a pattern.  Magnitudes have big impact when they are very large, but the same cannot be said of patterns.  The impact of pattern changes is much more complicated. (More on this in other posts in this series.)  Being a magnitude, the extreme outcome is can be unequivocally bad (electrical system blackout), unequivocally good (a hit movie), or both (some financial markets).

Second, the distribution of the generating process is stationary, means that it doesn't change it's structure over long time periods. In simple language, there is no "funny business" where some weird probability distribution takes over for a short period of time.  No space aliens.  No magic.  In a later post, I'll describe a different Swan that features non-stationary distributions.

Third, the generating process of a Grey Swan has unique characteristics:  at any point in time, there's a non-negligable probability of any outcome under the distribution (a.k.a. in the "support" of the distribution), even extreme values.  It's the combination of "any point in time" and "any outcome... even extreme values" that gives the Grey Swan its extreme impact and potential for surprise.  There will be other Shades of Swans that have a generating process whose outcomes are cumulative values that follow a Power Law function (e.g. Moore's Law and preferential attachment, a.k.a. "the rich get richer").

Fourth, the tail of the probability distribution of outcomes is heavy tailed, perhaps even a Pareto Distribution (a.k.a. power law).  For more on why this is, see my recent post: "How Fat-Tailed Probability Distributions Defy Common Sense and How to Handle Them".

Fifth, in many cases the evidence is limited to a comparatively small set of historical data.  For example, if we are trying to estimate the probability of a 10,000-year flood, we probably need much more than 50 or 100 years of data.

Finally, the primary formal reasoning tool is frequentist statistics where the past history is used to estimate the distribution of future events.  Beyond formal methods, it is often the case that informal reasoning dominates and thus the "Fast Thinking" system -- instinctive, experiential, emotional -- will be used to form expectations based on experience, stories, and social/cultural interactions.  This system is no good for anticipating or estimating stochastic processes with very heavy tails.

Examples

There are many examples of systems with generating processes that produce heavy tailed distributions in magnitudes, including:

• Earthquakes 
• Flood severity
• Hurricanes
• Financial market returns -- esp. markets prone to bubbles and crashes
• Internet traffic -- distribution of file sizes
• Sales of popular books, music, and movies
• Electric network blackout severity

However, not all of these systems suffer the other two features of Grey Swans -- 2) limited evidence and 3) frequentist statistics as the only or primary method of reasoning.  For natural disasters such as earthquakes, hurricanes, and floods, we've known for a long time that these had power law distributions in severity, so seeing very severe events can't be called "completely surprising".   Likewise, for Internet traffic, there is a very large volume of data so the distribution of file sizes can be estimated with good reliability.

Why Grey Swans Can Be Extreme

Grey swans have the potential for extreme consequences because small trigger events can "snowball" or cascade, and thereby draw large amounts of energy from a very large substrate system.

All Grey Swans draw their energy from a large or very large substrate system -- the earth's crust in the case of earthquakes, the securities markets in the case of financial bubbles and crashes, the national electrical grid in the case of blackouts.  Very often the system is a network that is prone to self-reinforcing cascades of interaction.  Furthermore, the system components are critically balanced (in a very technical sense of that term -- i.e at an unstable equilibrium).  They might be highly optimized in the sense that there is little or no redundancy, slack, delay, and also high interdependency between components.  (For more on man-made systems with highly optimized tolerances (HOT) see this, this and this.)

It is arguable whether the public Internet qualifies as a Grey Swan substrate system.  On the one hand it's very large and it's networked.  But it's not critically balanced.  There are plenty of redundancies and slack.

Let's consider which systems do not qualify under this definition of substrate system:

• Nearly all formal and informal organizations, taken individually
• Family and kinship structures, clans
• Towns, cities, states, and countries as political and social institutions

These systems do not qualify either because they are not big enough, not networked enough, lack tight interactions and cascades, and not critically balanced.  Instead, they are fairly robust to disturbances.

In contrast, large mobs can often cascade into "riots", so I think they do qualify as a viable substrate system for Grey Swans.

At a detailed level, there appear to be many possible mechanisms of how events cascade or "snowball" in these systems.  Power Law mechanisms in general is an open research problem. In this technical paper, J. Doyne Farmer and John Geanakoplos present and discuss nine alternative mechanisms in financial markets:

1. Hierarchies and exponentials 
2. Maximization principles 
3.  Maximum entropy 
4. Multiplicative processes 
5. Mixtures of distributions 
6. Preferential attachment 
7.  Dimensional constraints 
8.  Critical points and deterministic dynamics 
9.  “Trivial”mechanisms

Bottom line: While it's not always easy to pin down what a Grey Swan is, it's easier to say what it is not.  If the phenomenon doesn't reside in this type of substrate system, and if it is not prone to cascading/compounding disruptions without bound, then it is not a Grey Swan!

Why Grey Swans Can Be Surprising

Grey Swans are only surprising if we can't estimate with sufficient confidence the probability distribution of frequency and severity.  They can also be surprising if we try to make decisions based on simple averages or "typical values" when instead we should make decisions based on severity at 90th, 95th, 99th, or even 99.9th quantiles of the distribution.  ("Mean" is the 50th quantile.)

The biggest surprise will occur of people believe that the severity distribution is thin-tailed when it is really thick tailed.  This is the basis for Taleb's "Mediocrastan vs. Extremestan" metaphor.

I've written a separate blog post in tutorial style to explain the difference between thin-tailed and fat-tailed distributions and what we can do about them to not be so surprised.

Very important: the phrase "estimate with sufficient confidence" is not the same as estimating with precision nor does it mean being able to predict when a Grey Swan event will occur or how severe it will be. This is one of the greatest misunderstandings of Grey Swans.  This misconception is often asserted by arch-critics of probabilistic risk assessment, and it convinces some people who otherwise might be open minded.

What is "sufficient confidence" means is determined by the system in question and the decisions we need to make.  Moreover, if we have methods to rapidly and continuously improve our estimates (a.k.a. data-driven learning), then it's not so serious that our initial estimates might have wide confidence bounds or simplifications.

Why Grey Swans Can Be Rationalized in Retrospect

After an extreme Grey Swan event, we can revise our reasoning model to shift from a thin-tailed distribution estimate to a heavy-tailed estimate, or from a heavy-tailed estimate to a better one.  We might also have to change criteria used in decisions, including precautions for lack of precision in our estimates.

How to Cope with Grey Swans

From an analysts point of view, here's how you can better cope with Grey Swans:

1. To the method of frequentist statistical analysis of historical data, add other methods and other data to shed light on the "fatness" of the distribution tail.  Simulations, laboratory experiments, and subjective probability estimates by calibrated experts are just three alternative methods that can fill in for the limitations of frequentist methods with limited sample data. 
2. Communicate and decide using quantiles, not the usually summary statistics mean, standard deviation, etc. If any summary statistics are used as decision criteria or in models, use quantiles. 
3. Balance cautiousness with expediency. With any limited sample of data, be careful about tossing out "outliers". But also avoid the opposite error of being too cautious. Not every stochastic process has a heavy tailed distribution, and even those that do aren't necessarily "very fat" like the Pareto Distribution.
4. Avoid any statistical methods that assume an underlying normal distribution or thin tails. There are other methods that make few or no assumptions about the underlying distribution. They aren't as powerful, aren't as well known, and have other assumptions, but they are still useful. 
5. Put in some effort to estimate the "fatness" of the tail, either parametrically or non-parametrically. Even a not-very-good fat tail model is much better than one based on thin tails. There are ways to test how good the alternative models are. In my opinion, the best academic paper on this is "Power-law distributions in empirical data".

Regarding how policy makers should cope with Grey Swans. that's a much bigger topic that I won't go into now.  But the my favorite themes are "robustness", "resilience", "agility", and "creative destruction".