Riblet#33: Of Blips and Busts

“The Curse of Sports Illustrated” is where a sports person is featured on the front cover of a prominent magazine, and then their performance immediately plummets. Conversely, football managers may get sacked after a run of defeats only to see their successor being praised when results return to normal (by definition, a run will always come to an end). The causes of such changes will almost certainly not have been the photo on the cover nor the arrival of the new manager but regression to the mean, reversion to normal. 

Change brings risk, and it is important to understand the change in order to understand the risk.  You can then ensure that the action taken to mitigate it is effective.  It may be just a blip, and what you do to address it may not be what returns the situation to normality.

Speed cameras tend to be put in places where a string of accidents have recently occurred.  When the accident rate goes down, the improvement is attributed to the installation of the speed cameras.  The way to know if this was actually the case would be to set up a control group of randomly-placed speed cameras.  Where this has been done and monitored, it has been estimated that two thirds of the reduction in accidents at a blackspot was simply due to reversion to normal.

How do you know when a change is just a random variation, and when it is a fundamental shift that needs to be reacted to?

Models can help.  They are representations of an area of activity but, like maps, they may not have sufficient detail to provide you with what you need.  Once a model is out of the hands of its creators who understand its limitations, they can become dangerous.  Complex models used to determine the risk in bundles of mortgages worked well in the long period of boom in house prices, and they assumed only a moderate correlation between mortgage failures.  When conditions changed, the models vastly underestimated the risks, something on which the blame has been pinned for the financial crisis of 2008-12.

As with all aspects of risk management, a thorough understanding is the key.

Data Source: The Art of Statistics – David Spiegelhalter

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close