We (and our investments) Regress to Mediocrity

(Here is an interactive app I created to help you understand the ideas expressed in this post. Take a look either before or after reading the post.

We all have days where we are much more productive than usual. At such times we feel good about ourselves and think that we are turning a new leaf in terms of our productivity. Overtime however, we often find ourselves reverting back to our standard level of productivity. This concept is called “Regression to the Mean” ((Also known as Regression to Mediocrity.)) and it is a rule in statistics and about how the world works.

This rule builds on what we discussed in the previous post about how many things in the natural world fit the Normal Distribution. Most of the members of the distribution will be in the middle, around the mean ((Also known as the arithmetic average–we will use average and mean interchangeably.)) and the minority will be in the edges, known as “the tails”, of the distribution. Let’s take a real world example. The distribution (on the right) is of the heights of people, the average person is 68.3 inches tall, this is indicated by the blue line in the center on the graph (this one is called a histogram). The majority of the population are somewhere around average indicated by the blue line. Very few were above 72 inches (6 feet) or below 65 inches (5.3 feet). ((Note that in this dataset 1.08 inches was added to female heights to even out gender differences.))

If one compares the heights of parents to that of their children one finds that parents who are very tall have children who are slightly shorter than themselves and parents who are short have children who are slightly taller than them. This makes sense intuitively because if tall parents always had children who were taller than them some part of the population would incrementally get taller until we had a population of giants. Similarly if short parents consistently had even shorter children we would end up with part of the population who get unendingly shorter. Neither of these happen in the real world. So Regression to the Mean tells us that any extreme occurrence will not be permanent. Over time it will revert back to the mean.

Another real life example of this is hedge fund and mutual fund managers. At any given time you will have some who beat the market and outperform the others. Yet, this rarely lasts. According to a New Yorker Magazine piece in 2014 a third of hedge funds fail in a three year period:

“Out of an estimated seventy-two hundred hedge funds in existence at the end of 2010, seven hundred and seventy-five failed or closed in 2011, as did eight hundred and seventy-three in 2012, and nine hundred and four in 2013.”

Thus, whilst some people can beat the market average some of the time, Regression to the Mean informs us that it is very rare for people to be able to do it consistently. This law also applies to sports as well many other fields.

In order to illustrate this idea using real data, I have created a simple online application that you can play around with to see how the heights of people regress to the mean. This app uses the well-known Galton dataset, collected in 1885, of nearly 900 pairs of parents and their children and shows that the children of tall parents are on average shorter than their parents and the children of short parents are on average taller than their parents.

See the app here.

Next post will build on this idea and will be about “The Law of Large Numbers.”