When Do Predictions Make Sense?

The great sage Yogi Berra is rumored to have said “Predictions are tough, especially about the future.”  As with many of his quotes, it resonates with us, by being both silly and profound at the same time.  Predicting the future is a notoriously difficult business.  That doesn’t stop an industry of pundits prognosticating about elections, sporting events and the future of technology.  While these pundits typically have a dismal record of accurate predictions, their popularity is driven by the human quest for certainty.

In the corporate world, the business of prediction is taken seriously.  Firms believe that by understanding markets, trends and world events, they can plan better and achieve a competitive advantage over rivals.  They use consultants and analysts to establish predictive models for decisions on budgeting, hiring and technology selections.

But as alluring as predictive capability may be, we must understand the inherent limitations in human prescience.  In his book Expert Political Judgment: How Good Is It? How Can We Know?, University of Pennsylvania researcher Philip Tetlock takes a highly critical view of punditry.  In this seminal work, Tetlock evaluated the predictive accuracy of 284 political experts.  During a 15 year period, Tetlock had the experts make over 27,000 predictions.  The novel aspect of Tetlock’s work was that he forced the pundits to make specific, verifiable forecasts.  The results were not pretty.

The experts did no better than software that used random numbers to simulate decisions.  Tetlock also created several computer models that used historical data to make predictions.  Several of those models surpassed the accuracy of even the best expert pundits.

In the Wisdom of Crowds, James Surowiecki makes a similar case for the inaccuracy of expert prediction.  In his book, he compares the expert’s forecasts to the averaged predictions of large groups.  Just as with computer models, he finds the crowd beats the experts.

So what are we to make of expert forecasting?  Should we simply dismiss it, putting it in the same domain as psychic predictions?  The key to understanding the relevance of forecasting is to recognize that events fall into one of two categories:

  • Simple events that can have assignable odds.
  • Complex events that don’t behave in predictable ways

A spin of a roulette wheel and dice throws are examples of deterministic events.  They are what probability experts would describe as well-behaved.  That is, their expected outcomes can be accurately stated with specific odds.  Given a large enough sample, simple events will tend to produce results that are highly consistent with their predicted outcomes.

Complex events, however, are highly unpredictable.  They have numerous underlying factors that interact in a multitude of complicated ways.  Some complex events are chaotic with major swings in outcome based on minor changes in underlying variables.  Examples of complex events are interest rate changes, new product introductions and political upheavals.  Trying to accurately forecast their impacts or outcomes is next to impossible.

Unfortunately, very few workplace events mirror the neat and tidy determinism of a roulette wheel.  Most are in fact the messy, complex sort that is so unpredictable.  Fortunately, there is a practical way to assess events to decide if their outcome can be accurately forecast.  Events will be more accurately predictable if they have the following properties:

  • Limited, definable set of potential outcomes
  • Limited, definable set of input factors
  • Large history of previous outcomes
  • Large sample set

An example of a reasonably predictable event would be the likelihood of failures of disk drives in a large storage farm.  Understanding this probability is helpful in deciding optimal RAID set sizes as well as maintenance practices.  It is reasonably predictable because it meets the criteria above:

  • Outcomes – limited to “working” or “failed”
  • Input Factors – Heat, Number of rotations, Number of power cycles are the main drivers of lifespan
  • History – There is significant historical data from the manufacturer (expressed in the form of Mean Time Between Failures)
  • Sample Set – There are typically hundreds to thousands of drives in a farm

Now let’s look at an example on the opposite end of the spectrum.  Let’s say a department selling disk drives to the healthcare industry has decided to raise the quotas for its 5 salespeople by 10% this quarter.  Can we easily predict the effect this will have on sales?  Let’s look at the same properties that we defined above:

  • Outcomes – Range from complete drop-off of sales to large increases
  • Input Factors – Reaction of sales staff, turnover, local job market, customer budgets, purchasing cycles, competitors, state of economy, legislation
  • History – Low.  Unlikely that similar quota changes were tried for these specific salespeople with identical input factors.
  • Sample Set – Small, just 5 salespeople, 1 product line, 1 customer vertical

It is unlikely that we will be able to predict the level of sales next quarter based on this 10% quota increase.  The number of complex and diverse input factors makes the exercise especially daunting.  The small sample size adds to the fragility of the forecasting exercise.  Let’s contemplate just a few factors that would be unpredictable and add havoc into our forecast:

  • A competitor introduces a new “killer” product at a much lower price point
  • New health care legislation is proposed, crimping your customer’s budgets
  • A surprise court ruling has customers scrambling to implement data archiving environments
  • A customer who is a strong advocate of your product line gets promoted into a position of purchasing authority
  • A highly heated local job market causes you to lose your star salesperson

I could go on for several pages with numerous additional factors.  It is clear that a firm can expect limited accuracy when predicting the outcome of these types of complex events.  To ensure the greatest success around forecasting, firms would be wise to adopt the following practices:

  • Use the set of properties above to decide what types of events are worth predicting.
  • Recognize that being an “expert” is not a formula for accurate forecasting.
  • Wherever possible, use the “wisdom of the crowds” model, getting input from a broad array of team members.  Consider employing internal Prediction Markets to harness this wisdom.
  • Recognize the inherent fallibility in the forecasting process.  Don’t jeopardize your business by counting on unreasonable accuracy.
This entry was posted in Complexity, Organization, Probability/Statistics. Bookmark the permalink.