A Grey Swan -- almost Black, but not quite. More narrowly defined. |

I'll define and describe each one, and maybe give some examples. Most important, each of these Shades will be defined by a mostly-unique set of 1)

*generating process(es)*; 2)

*evidence and beliefs*; and 3)

*methods of reasoning and understanding*. As described in the introductory post, it's only in the interaction of these three that Black Swan phenomena arise. Each post will close with section called "How To Cope..." that, hopefully, will make it clear why this Many Shades approach is better than the all-lumped together Black Swan category.

This first one is named "Grey" because it's closest to Taleb's original concept before it got hopelessly expanded and confused.

### Definition

A "Grey Swan" as a stochastic process and estimation system where:- The
*generating process*is effectively a stationary heavy-tailed distribution of magnitudes; - The
*evidence*is a comparatively small sample of history, with the taken-for-granted*belief*that the distribution is thin-tailed (i.e. not heavy-tailed); - The
*method of reasoning*is frequentist statistics, using only available history and maybe also statistical methods that assume thin-tailed distributions.

"H T T H T T H T T T T H H H T H H T T H H T...".Likewise, the output of a stochastic process could be a series of magnitudes (numbers), a series of objects, or a series of patterns, each drawn from a probability distribution of outcomes.

### Main Features

First, it's important to note that what is being generated is a*not a pattern. Magnitudes have big impact when they are very large, but the same cannot be said of patterns. The impact of pattern changes is much more complicated. (More on this in other posts in this series.) Being a magnitude, the extreme outcome is can be unequivocally bad (electrical system blackout), unequivocally good (a hit movie), or both (some financial markets).*

**magnitude**,Second, the distribution of the generating process is

*, means that it doesn't change it's structure over long time periods. In simple language, there is no "funny business" where some weird probability distribution takes over for a short period of time. No space aliens. No magic. In a later post, I'll describe a different Swan that features non-stationary distributions.*

**stationary**Third, the generating process of a Grey Swan has unique characteristics:

*at*(a.k.a. in the "support" of the distribution),

**any point in time**, there's a non-negligable probability of**any****outcome**under the distribution*. It's the combination of "any point in time" and "any outcome... even extreme values" that gives the Grey Swan its extreme impact and potential for surprise. There will be other Shades of Swans that have a generating process whose outcomes are*

**even extreme values***cumulative values*that follow a Power Law function (e.g. Moore's Law and preferential attachment, a.k.a. "the rich get richer").

Fourth, the tail of the probability distribution of outcomes is

*, perhaps even a Pareto Distribution (a.k.a. power law). For more on why this is, see my recent post: "How Fat-Tailed Probability Distributions Defy Common Sense and How to Handle Them".*

**heavy tailed**Fifth, in many cases the evidence is

*limited to a comparatively small set of historical data*. For example, if we are trying to estimate the probability of a 10,000-year flood, we probably need much more than 50 or 100 years of data.

Finally, the primary formal reasoning tool is

*frequentist statistics*where the past history is used to estimate the distribution of future events. Beyond formal methods, it is often the case that

*informal reasoning dominates*and thus the "Fast Thinking" system -- instinctive, experiential, emotional -- will be used to form expectations based on experience, stories, and social/cultural interactions. This system is no good for anticipating or estimating stochastic processes with very heavy tails.

### Examples

There are many examples of systems with generating processes that produce heavy tailed distributions in magnitudes, including:

- Earthquakes
- Flood severity
- Hurricanes
- Financial market returns -- esp. markets prone to bubbles and crashes
- Internet traffic -- distribution of file sizes
- Sales of popular books, music, and movies
- Electric network blackout severity

### Why Grey Swans Can Be Extreme

Grey swans have the potential for extreme consequences because small trigger events can "snowball" or cascade, and thereby draw large amounts of energy from a very large substrate system.

All Grey Swans

All Grey Swans

*draw their energy from a large or very large*-- the earth's crust in the case of earthquakes, the securities markets in the case of financial bubbles and crashes, the national electrical grid in the case of blackouts. Very often the system is a network that is**substrate system***prone to self-reinforcing cascades of interaction*. Furthermore, the*system components are critically balanced*(in a very technical sense of that term -- i.e at an unstable equilibrium). They might be*highly optimized*in the sense that there is little or no redundancy, slack, delay, and also high interdependency between components. (For more on man-made systems with highly optimized tolerances (HOT) see this, this and this.)It is arguable whether the public Internet qualifies as a Grey Swan substrate system. On the one hand it's very large and it's networked. But it's not critically balanced. There are plenty of redundancies and slack.

Let's consider which systems

*do not qualify*under this definition of substrate system:- Nearly all formal and informal organizations, taken individually
- Family and kinship structures, clans
- Towns, cities, states, and countries as political and social institutions

These systems do not qualify either because they are not big enough, not networked enough, lack tight interactions and cascades, and not critically balanced. Instead, they are fairly robust to disturbances.

In contrast, large mobs can often cascade into "riots", so I think they do qualify as a viable substrate system for Grey Swans.

In contrast, large mobs can often cascade into "riots", so I think they do qualify as a viable substrate system for Grey Swans.

At a detailed level, there appear to be many possible mechanisms of how events cascade or "snowball" in these systems. Power Law mechanisms in general is an open research problem. In this technical paper, J. Doyne Farmer and John Geanakoplos present and discuss nine alternative mechanisms in financial markets:

- Hierarchies and exponentials
- Maximization principles
- Maximum entropy
- Multiplicative processes
- Mixtures of distributions
- Preferential attachment
- Dimensional constraints
- Critical points and deterministic dynamics
- “Trivial”mechanisms

__Bottom line:__While it's not always easy to pin down what a Grey Swan is, it's easier to say

*what it is not.*If the phenomenon doesn't reside in this type of substrate system, and if it is not prone to cascading/compounding disruptions without bound,

*then it is not a Grey Swan*!

### Why Grey Swans Can Be Surprising

Grey Swans are only surprising if we can't estimate with sufficient confidence the probability distribution of frequency and severity. They can also be surprising if we try to make decisions based on simple averages or "typical values" when instead we should make decisions based on severity at 90th, 95th, 99th, or even 99.9th quantiles of the distribution. ("Mean" is the 50th quantile.)

The biggest surprise will occur of people believe that the severity distribution is thin-tailed when it is really thick tailed. This is the basis for Taleb's "Mediocrastan vs. Extremestan" metaphor.

I've written a separate blog post in tutorial style to explain the difference between thin-tailed and fat-tailed distributions and what we can do about them to not be so surprised.

What is "sufficient confidence" means is determined by the system in question and the decisions we need to make. Moreover, if we have methods to rapidly and continuously improve our estimates (a.k.a. data-driven learning), then it's not so serious that our initial estimates might have wide confidence bounds or simplifications.

The biggest surprise will occur of people believe that the severity distribution is thin-tailed when it is really thick tailed. This is the basis for Taleb's "Mediocrastan vs. Extremestan" metaphor.

I've written a separate blog post in tutorial style to explain the difference between thin-tailed and fat-tailed distributions and what we can do about them to not be so surprised.

**: the phrase "estimate with sufficient confidence" is**__Very important__*not the same as estimating with precision**nor does it mean being able to predict when a Grey Swan event will occur or how severe it will be*. This is one of the greatest misunderstandings of Grey Swans. This misconception is often asserted by arch-critics of probabilistic risk assessment, and it convinces some people who otherwise might be open minded.What is "sufficient confidence" means is determined by the system in question and the decisions we need to make. Moreover, if we have methods to rapidly and continuously improve our estimates (a.k.a. data-driven learning), then it's not so serious that our initial estimates might have wide confidence bounds or simplifications.

### Why Grey Swans Can Be Rationalized in Retrospect

After an extreme Grey Swan event, we can revise our reasoning model to shift from a thin-tailed distribution estimate to a heavy-tailed estimate, or from a heavy-tailed estimate to a better one. We might also have to change criteria used in decisions, including precautions for lack of precision in our estimates.### How to Cope with Grey Swans

From an analysts point of view, here's how you can better cope with Grey Swans:

- To the method of frequentist statistical analysis of historical data,
*add other methods and other data to shed light on the "fatness" of the distribution tail.*Simulations, laboratory experiments, and subjective probability estimates by calibrated experts are just three alternative methods that can fill in for the limitations of frequentist methods with limited sample data. *Communicate and decide using quantiles*, not the usually summary statistics mean, standard deviation, etc. If any summary statistics are used as decision criteria or in models, use quantiles.*Balance cautiousness with expediency*. With any limited sample of data, be careful about tossing out "outliers". But also avoid the opposite error of being too cautious. Not every stochastic process has a heavy tailed distribution, and even those that do aren't necessarily "very fat" like the Pareto Distribution.*Avoid any statistical methods that assume an underlying normal distribution or thin tails.*There are other methods that make few or no assumptions about the underlying distribution. They aren't as powerful, aren't as well known, and have other assumptions, but they are still useful.*Put in some effort to estimate the "fatness" of the tail*, either parametrically or non-parametrically. Even a not-very-good fat tail model is much better than one based on thin tails. There are ways to test how good the alternative models are. In my opinion, the best academic paper on this is "Power-law distributions in empirical data".

Regarding how policy makers should cope with Grey Swans. that's a much bigger topic that I won't go into now. But the my favorite themes are "robustness", "resilience", "agility", and "creative destruction".

## No comments:

## Post a Comment