School of Psychological Science, La Trobe University, Australia.
Reducing Overconfidence in the Interval Judgments of Experts
Article first published online: 17 DEC 2009
© 2009 Society for Risk Analysis
Volume 30, Issue 3, pages 512–523, March 2010
How to Cite
Speirs-Bridge, A., Fidler, F., McBride, M., Flander, L., Cumming, G. and Burgman, M. (2010), Reducing Overconfidence in the Interval Judgments of Experts. Risk Analysis, 30: 512–523. doi: 10.1111/j.1539-6924.2009.01337.x
- Issue published online: 8 MAR 2010
- Article first published online: 17 DEC 2009
- Expert elicitation;
- interval judgment;
Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3-point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4-step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta-analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)—a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4-step procedure is more likely to reduce overconfidence than the 3-point procedure (Cohen's d = 0.61, [0.04, 1.18]).