A quantitative assessment of the quality of strategic intelligence forecasts / by David R. Mandel, Alan Barnes and Karen Richards.: D68-4/036-2013E-PDF

This report describes a field study of the quality of probabilistic forecasts made in Canadian strategic intelligence reports. The researchers isolated a set of 1,422 probabilistic forecasts from intelligence memoranda and interdepartmental committee reports for which outcome information about the forecasted events was available. These data were used to study forecast quality measures, including calibration and discrimination indices, commonly employed in other areas of expert judgment monitoring research (e.g., meteorology or medical diagnosis). Predictions were further categorized in terms of other variables, such as the organizational source, forecast difficulty, and forecast importance. Overall, the findings reveal a high degree of forecasting quality. This was evident in terms of calibration, which measures the concordance between probability levels assigned to forecasted outcomes and the relative frequency of observed outcomes within that assigned category. It was also evident in terms of adjusted normalized discrimination, which measures the proportion of outcome variance explained by analysts’ forecasts. The main source of bias detected in analytic forecasts was underconfidence: Analysts often rendered forecasts with greater degrees of uncertainty than were warranted. Implications for developing outcome-oriented accountability systems, adaptive learning systems, and forecast optimization procedures to support effective decision-making are discussed.

Permanent link to this Catalogue record:
publications.gc.ca/pub?id=9.821111&sl=0

• MARC XML format
• MARC HTML format

Request alternate formats