Ending the Illusion of Control: Let’s Kick This Bad Forecasting Habit, Spyros Makridakis, Robin M Hogarth & Anil Gaba

The failure of recent forecasts.

Many reputable institutions issue regular economic forecasts that are widely cited in the international

press. And yet, how accurate are these forecasts? Should we really trust them?

In April 2008, for example, the International Monetary Fund (IMF) projected that global economic

growth would slow to 3.7 percent in 2008, and remain broadly unchanged in 2009. In addition, it

predicted that the U.S. economy would tip into a mild recession in 2008 as a result of mutually

reinforcing cycles in the housing and financial markets, before starting a modest recovery in 2009.

In October 2008, the IMF stated that the world economy was entering a major downturn and that

economic growth would slow substantially in 2009 with a modest recovery to begin only later in the

year. In January, 2009 the growth forecast was cut to 0.5%. In February it was further reduced to

0.0%. And in March it slid into the negative for the first time since the 1940s.

Now, the IMF is an international organization employing hundreds of economists, including some of

the best and brightest in the world. How could it be so wrong in forecasting a 3.7% growth in 2009

when the economy was already in recession? How could it then predict that the US was facing a mild

recession instead of a meltdown? And how could it change its forecast from a healthy 3.7% to negative

growth in less than ten months?

In this manifesto, we note the dismal historical track record of much if not most economic forecasting.

Despite this reality, however, people still have an insatiable need to issue and consume forecasts.

We therefore make three proposals that can make forecasts both more relevant and useful.

The inaccuracy and uncertainty of forecasts.

Influenced by the stagflation prevailing in the latter part of the 1980s, several books predicted

a serious economic downturn or even depression for the 1990s. The following is a quote from one

of these books, The Great Depression of 1990 by Ravi Batra:

“I am an economist, trained in scientific analysis, not a sensationalist

or a Jeremiah. Yet all the evidence indicates that another great depression

is now in the making, and unless we take immediate action the price we

will have to pay in the 1990s is catastrophic.”

Yet the 1990s saw one of the biggest booms in world history. The Dow Jones Industrial Average (DJIA)

share index grew from 2,753 at the beginning of the decade to 11,358 at the end, a 313% increase.

By the end of 1990s, the climate had changed to one of jubilation with books predicting that

the DJIA would reach 36,000, 40,000, or even 100,000. One book, Dow 36,000, was upfront with

that prediction:

“A sensible target date for Dow 36,000 is early 2005, but it could be reached much

earlier. After that, stocks will continue to rise, but at a slower pace. This means

that stocks, right now, are an extraordinary investment. They are just as safe as

bonds over long periods of time, and the returns are significantly higher.”

On March 9, 2009, not only had the DJIA failed to reach 36,000 but it had fallen to close to 6,547—

well below its value in 1999 when Dow 36,000 was published. Will the Dow ever reach 36,000?

Perhaps, given enough time, but it is certain that nobody can predict when.

Another book, The Roaring 2000s, published in 1998, confidently announces an “unprecedentedAnd there is even better news:

stock boom that almost none of our economists forecast.”

“It will continue until about mid-2009, and the Dow will reach at least 21,500 and likely

higher.”

Yet the same author published a new book in January 2009, in the middle of a major

recession, entitled The Great Depression Ahead, and completely ignored his own previous forecasts.

The inaccuracy of forecasting extends beyond the stock market. Consider for instance the oil price

forecast by Goldman Sachs, the world’s biggest trader in oil futures. In March 2008, when oil prices

were increasing rapidly, it predicted that $200 a barrel was not very far off. Subsequently, as oil

prices started declining, Goldman Sachs cut its forecasts to $80 and then, in December 2008, to $45

a barrel. Interestingly, this is a case of history repeating itself. In the early 1980s there were forecasts

that the price of oil would jump to $250 (in constant 2008 prices) a barrel while, in fact, it dropped

to as little as $15 a barrel before the end of the decade.

What about predictions in the real estate market? The title of a book published in 2005 says it all:

Why the Real Estate Boom Will Not Bust—And How You Can Profit from It: How to Build Wealth in

Today’s Expanding Real Estate Market. Needless to say, in March 2009, the average drop in house

prices was around 30% from its peak in the USA, while in the UK a leaked Numis Securities report

estimates that house prices could fall by a further 55%, saddling millions with properties worth

less than their mortgages.

In the 1970s and 1980s, social scientists had hoped that computers and sophisticated mathematical

models would allow them to emulate the forecasting successes of the hard sciences. Although

only the tip of the iceberg, examples such as the above have revealed their hopes to be groundless.

Forecasting accuracy is low in the great majority of areas affecting our lives and work. In addition,

predictions can change considerably from one forecaster to another and across time, thereby

increasing uncertainty and raising serious questions about the relevance and practical value of

forecasting. The big question is what to do to improve accuracy and to make forecasts more relevant

and useful to decision and policy makers. “Scientific” forecasting must distinguish itself from

astrology and provide value instead of big surprises.

The empirical evidence.

Unlike astrology, the field of forecasting has progressed considerably over the last four decades

through the accumulation of a large body of empirical evidence. However, we stand in serious danger

of failing to profit from this evidence. Some forecasters either choose to ignore it or, worse, deliberately

withhold data that threatens their reputations. At the same time, however, there are responsible

forecasters who are using the evidence to learn how to improve their accuracy and to assess future

uncertainty more realistically.

Below is a summary of the empirical evidence collected through many studies and subsequently

replicated by others.

• The future is never exactly like the past. This means that the extrapolation of past patterns

or relationships (which is current practice) cannot provide accurate predictions.

• Statistically sophisticated or complex models fit past data well, but do not necessarily

predict the future accurately.

• “Simple” models do not necessarily fit past data well, but predict the future better than

complex statistical models.

• Statistical models underestimate uncertainty, sometimes enormously, because they more

often than not assume that:

a. Events are independent:

This is a serious mistake as the world has become

a global village. What happens in one place (e.g. the toxic mortgages in the USA)

seriously affects the entire world.

b. Forecasting errors are manageable:

This is not true in the great majority of

financial and business situations, which are characterized by extreme and erratic

errors of prediction. The result is that models underestimate the volatility of economic

phenomena and, in particular, fail to provide early warnings of cataclysmic events.

c. The dispersion of forecasting errors is finite, known and constant:

It is not.

For example, stock market volatility was considerably higher in 2008 than in previous

years. The result is much greater uncertainty and many surprises.

• Humans, driven by over-optimism and wishful thinking, often underestimate uncertainty

even more than statistical models do.

• Events that are not only unforeseen but unforeseeable occur frequently. These are what

Nassim Nicholas Taleb eloquently refers to as “Black Swans.”

• Expert judgment is typically inferior to simple statistical models.
• Forecasts made by experts are no more accurate than those of knowledgeable individuals.
• Averaging the predictions of several individuals usually improves forecasting accuracy.
• Averaging the forecasts of two or more models improves accuracy while also reducing the

dispersion of forecasting errors.

Three proposed changes.

1. Dispel the illusion of accurate forecasts:

The time has come to recognize that no one has a

crystal ball. To believe otherwise is not only illusory, but also potentially dangerous. Such recognition

is a prerequisite for developing strategies to cope with low predictability and prepare for unanticipated

events. Venture capitalists, for instance, accept that they cannot predict successful start-ups.

Instead, they hedge their bets across a broad range of investments while concentrating on producing

a big success to overcompensate for the failures. Similarly, when we buy insurance we accept our

inability to predict car accidents, fires or floods. The same is true with reserves for bad debts, which

acknowledge our inability to predict when such debts will occur. Why not build similar reserves for

times of recession, which—we now know—are equally hard to predict?

2. Concentrate on uncertainty:

We can always make a prediction, either judgmentally or using a

statistical model. Once such a forecast exists, the more difficult task is to assess its accuracy and the

resulting uncertainty. Rare events or “Black Swans” pose a particular challenge. It is therefore tempting

to ignore them or treat them as outliers. To deal with these problems, the authors have proposed

the “three A’s” approach for dealing with uncertainty realistically. This starts with (1) Accepting that

uncertainty exists, (2) Assessing its level objectively using past information, as well as alternative

future scenarios and, finally (3) Augmenting the uncertainty so it’s assessed to account for the

fact that both models and humans tend to underestimate it—in other words, to compensate for our

limited imaginations.

3. Evaluate forecasters, and hold them responsible:

At present, anyone can make forecasts

without providing indications of possible inaccuracy or the underlying assumptions made. We

propose that, as a minimum, all predictions should be accompanied by the following information:

a.

A clear list of all assumptions made.

b.

An explanation of how the forecast will change if one or more of these assumptions

turn out to be wrong.

c.

In addition to the single main forecast, a range of values showing the uncertainty involved,

together with an indication of the likelihood that even these ranges might be exceeded.

d.

For organizations issuing regular forecasts, a statement of their past accuracy, using

simple benchmarks and even comparisons with simple alternative forecasting models.

e.

A reminder, similar to those on cigarette packages, stating the dangers of inaccurate

forecasts.

In the end, the comparison with smoking seems particularly apt. As a race, we humans seem to

have developed a pretty bad forecasting dependency. It is time to kick this dangerous habit,

before it inflicts further damage on our economic health.

About the Authors

Spyros Makridakis

is founding chief editor of the Journal of Forecasting and the International Journal

of Forecasting and Professor Emeritus at INSEAD. Robin M Hogarth (formerly of the University of Chicago)

is now ICREA Research Professor at Universitat Pompeu Fabra, Barcelona and President of the European

Association for Decision Making. Anil Gaba directs the Centre on Decision Making and Risk Analysis at

INSEAD where he is the ORPAR Chaired Professor of Risk Management. Together they are the coauthors of

the new book Dance With Chance: Making Luck Work For You (Oneworld Publications).

Source: Change This