• Applying numbers to the innumerable

    Ben D. Kritz

    Ben D. Kritz

    EARLIER this week I read an enlightening essay by Stratfor’s Dan Gardner and Philip E. Tetlock, who proposed an idea so unusual it’s almost sacrilegious.

    “One simple change to forecasters’ standard operating procedure could boost forecast accuracy, increase accountability, reduce misunderstandings and miscalculations, and generally make the world a wealthier and safer place,” they write.

    “The change? Use numbers. No more saying ‘it is likely,’ or ‘improbable,’ or ‘to be expected’ or ‘all but certain.’ Instead, say there is a 60 percent, 23 percent, 78 percent or 95 percent chance. That’s it. If pundits, journalists, economists, intelligence analysts, geo-strategists and others who prognosticate for a living switch to numbers, they would do nothing less than improve humanity’s collective foresight.”

    The heart of their argument is that most forecasting is not necessarily presented as such, and the reliance on language for making predictive judgments leads to misunderstanding, because language is ambiguous. For example, a statement such as, “The British pound may collapse” (an example the authors use) actually means nothing by itself. It may collapse; it may not. The statement requires context; supporting information to further define what “may” actually means, or as the authors explain, if the statement is being delivered verbally, body language and tone to help the audience decide what prediction is being made.

    It’s easy to see where the problems lie with that sort of forecasting. From the perspective of the audience, the statement does not provide guidance as to how they should react; it leaves them with the questions, “What does ‘collapse’ mean in tangible terms, and how likely is that to happen?” They use other cues – the authority of the source, and supporting statements that led the source to draw that particular ambiguous conclusion – to answer those questions for themselves, and in doing so, do not critically consider how empty the statement really is.

    From the perspective of the forecaster, it means that, given the uncritical nature of the audience, he will be correct, albeit in a terribly lazy way, no matter what the actual outcome. If the pound truly collapses and loses half its value in a single day, he can later say, “I told you so.” If the pound declines a few pence against the euro or the dollar, that might be enough to define a “collapse,” or as an alternative, illustrate the circumstances putting the pound at risk, which, after all, was an underlying message of the prediction. If nothing untoward happens, then the conclusion is, “There was a possibility the pound could collapse, which fortunately did not come to pass because this or that did or did not happen.”

    To put it plainly, a “prediction” of that nature accomplishes precisely nothing, and yet most forecasting – certainly in economics, and in other disciplines that rely heavily on qualitative judgments as well – is based on those sorts of statements. Obviously, Gardner and Tetlock think that is wrong, although it must be acknowledged that applying a strictly quantitative approach is not universally appropriate, either; we have the examples of various models applied to stock markets that have not worked as well in the real world as they do on paper as evidence of that.

    Rather, the real point is that would-be forecasters – present company included – ought to strive to apply greater precision to predictions. Trying to think of an economic problem like “Is the pound going to collapse?” in empirical terms forces one to consider in fine detail the various factors that may be relevant, and consider a broader range of factors.

    For instance, we might discover, after examining the issue as thoroughly as possible, that there are 100 different individual factors that we can identify as having a “yes-no” or “positive-negative” impact on the up-or-down direction of the pound’s value. By assessing each of those factors individually and making a series of small judgments (we economists don’t like to use the word “guess”), we can develop an actual probability estimate – one that may still be largely based on qualitative conclusions, but can be expressed in numerical terms, such as “there is a 38 percent chance that the pound will decline by more than 10 pence against the dollar.” How far the actual outcome diverges from that prediction gives us a better idea of how our forecasting can be improved. From the perspective of the audience, the deviation serves as a scorecard for which sources of forecasts are better than others.

    Political and other social analysts may find the numeric way of thinking a little more difficult to grasp, but in terms of economics, it should be a natural fit. After all, our interest in the discipline assumes a rather high level of comfort with numbers. We ought to make more use of them.



    Please follow our commenting guidelines.

    Comments are closed.