Research Article

You have full text access to this OnlineOpen article

Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation

  1. S. J. Mason*,
  2. N. E. Graham

Article first published online: 29 DEC 2006

DOI: 10.1256/003590002320603584

Issue

Quarterly Journal of the Royal Meteorological Society

Quarterly Journal of the Royal Meteorological Society

Volume 128, Issue 584, pages 2145–2166, July 2002 Part B

Additional Information

How to Cite

Mason, S. J. and Graham, N. E. (2002), Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Q.J.R. Meteorol. Soc., 128: 2145–2166. doi: 10.1256/003590002320603584

Author Information

  1. Scripps Institution of Oceanography, USA

*Climate Research Division, Scripps Institution of Oceanography, University of California San Diego, La Jolla, CA 92093-0230, USA.

Publication History

  1. Issue published online: 29 DEC 2006
  2. Article first published online: 29 DEC 2006
  3. Manuscript Revised: 23 APR 2002
  4. Manuscript Received: 5 NOV 2001

SEARCH

SEARCH BY CITATION

ARTICLE TOOLS

Get PDF (356K)

REFERENCES

  • Bamber, D. 1975 The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J. Math. Psychol., 12, 387–415
  • Begg, C. B. 1991 Advances in statistical methodology for diagnostic medicine in the 1980s. Stat. Med., 10, 1887–1895
    Direct Link:
  • Buckle, N., Kraft, C. H. and van Eeden, C. 1969 An approximation to the Wilcoxon–Mann–Whitney distribution. J. Am. Stat. Soc., 64, 591–599
  • Buizza, R. 2001 Accuracy and potential economic value of categorical and probabilistic forecasts of discrete events. Mon. Weather Rev., 129, 2329–2345
  • Buizza, R. and Palmer, T. N. 1998 Impact of ensemble size on ensemble prediction. Mon. Weather Rev., 126, 2503–2518
  • Buizza, R., Petroliagis, T., Palmer, T. N., Barkmeijer, J., Hamrud, M., Hollingsworth, A., Simmons, A. and Wedi, N. 1998 The impact of model resolution and ensemble size on the performance of an ensemble prediction system. Q. J. R. Meteorol. Soc., 124, 1935–1960
    Direct Link:
  • Buizza, R., Hollingsworth, A., Lalaurette, E. and Ghelli, A. 1999 Probabilistic predictions of precipitation using the ECMWF ensemble prediction system. Weather and Forecasting, 14, 168–189
  • Campbell, G. 1994 Advances in statistical methodology for the evaluation of diagnostic and laboratory tests. Stat. Med., 13, 499–508
    Direct Link:
  • Centor, R. M. 1991 Signal detectability: the use of ROC curves and their analyses. Med. Decision Making, 11, 102–106
  • Centor, R. M. and Schwartz, J. S. 1985 An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve. Med. Decision Making, 15, 276–282
  • Conover, W. J. 1973 Rank tests for one sample, two samples, and k samples without the assumption of a continuous distribution function. Ann. Stat., 1, 1105–1125
  • 1999 Practical nonparametric statistics. Wiley, Chichester, UK
  • DeLong, E. R., DeLong, D. M. and Clarke-Pearson, D. L. 1988 Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44, 837–845
  • Dineen, L. C. and Blakesley, B. C. 1973 Algorithm AS 62. A generator for the sampling distribution of the Mann–Whitney U statistic. Appl. Stat., 22, 268–273
  • Dodge, H. and Romig, H. 1929 A method of sampling inspection. Bell Systems Tech. J., 8, 613–631
  • Dooley, K. 2000 The paradigms of quality: evolution and revolution in the history of the discipline. Adv. Manage. Organ. Qual., 5, 1–28
  • Dorfman, D. D. and Alf, E. 1969 Maximum likelihood estimation of parameters of signal-detection theory and determination of confidence intervals–rating-method data. J. Math. Psychol., 6, 487–496
  • Egan, J. P. 1975 Signal detection theory and ROC analysis. Academic Press, New York, USA
  • Falmagne, J.-C. 1985 Elements of psychophysical theory. Oxford University Press, Oxford, UK
  • Frogner, I.-L. and Iversen, T. 2001 Targeted ensemble prediction for northern Europe and parts of the North Atlantic Ocean. Tellus, 53A, 35–55
  • Gail, M. H. and Green, S. B. 1976 A generalization of the one-sided two-sample Kolmogorov–Smirnov statistic for evaluating diagnostic tests. Biometrics, 32, 561–570
  • Graham, N. E. 1994 ‘Experimental predictions of wet season precipitation in northeastern Brazil’. Pp. 378–381 in Proceedings of the 18th Annual Climate Diagnostics Workshop, 1–5 November 1993, Boulder, Colorado, USA
  • Graham, R. J., Evans, A. D. L., Mylne, K. R., Harrison, M. S. J. and Robertson, K. B. 2000 An assessment of seasonal predictability using atmospheric general circulation models. Q. J. R. Meteorol. Soc., 126, 2211–2240
  • Green, D. M. and Swets, J. A. 1966 Signal detection theory and psychophysics. Peninsula Publishing, Los Altos, California, USA
  • Grey, D. R. and Morgan, B. J. T. 1972 Some aspects of ROC curve-fitting: normal and logistic models. J. Math. Psychol., 9, 128–139
  • Hanley, J. A. 1988 The robustness of the ‘binormal’ assumptions used in fitting ROC curves. Med. Decision Making, 8, 197–203
  • Hanley, J. A. and McNeil, B. J. 1982 The meaning and use of the area under the receiver operating characteristic (ROC) curve. Radiology, 143, 29–36
  • 1983 A method of comparing the areas under receiver operating characteristic curves from the same cases. Radiology, 148, 839–843
  • Harding, D. E. 1984 An efficient, minimal-storage procedure for calculating the Mann–Whitney U, generalized U and similar distributions. Appl. Stat., 33, 1–6
  • Harvey, L. O., Hammond, K. R., Lusk, C. M. and Mross E. F. 1992 The application of signal detection theory to weather forecasting behavior. Mon. Weather Rev., 120, 863–883
  • Hilden, J. 1991 The area under the ROC curve and its competitors. Med. Decision Making, 11, 95–101
  • Hoeffding, W. 1948 A class of statistics with asymptotically normal distribution. Ann. Math. Stat., 19, 293–325
  • Hsieh, F. and Turnbull, B. W. 1996 Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann. Stat., 24, 25–40
  • Kendall, M. G. and Stuart, A. 1977 The advanced theory of statistics. Griffin, London, UK
  • Klotz, J. 1966 The Wilcoxon, ties, and the computer. Ann. Math. Stat., 61, 772–787
  • Knuth, D. E. 1998 The art of computer programming. Volume 3: Sorting and searching. Addison–Wesley, Reading, Massachusetts, USA
  • Lovell, D. R., Dance, C. R., Niranjan, M., Prager, R. W. and Dalton, K. J. 1996 ‘Ranking the effect of different features on the classification of discrete valued data’. Pp. 487–494 in Proceedings of the second international conference on engineering applications of neural networks, 17–19 June 1996, Kingston upon Thames, London, UK
  • McClish, D. K. 1987 Comparing the areas under more than two independent ROC curves. Med. Decision Making, 7, 149–155
  • Mann, H. B. and Whitney, D. R. 1947 On a test of whether one or two random variables is stochastically larger than the other. Ann. Math. Stat., 18, 50–60
  • Mann, F. A., Hildebolt, C. F. and Wilson, A. J. 1992 Statistical analysis with receiver operating characteristic curves. Radiology, 184, 37–38
  • Mason, I. 1979 On reducing probability forecasts to yes/no forecasts. Mon. Weather Rev., 107, 207–211
  • 1982 A model for assessment of weather forecasts. Aust. Meteorol. Mag., 30, 291–303
  • Mason, S. J. and Graham, N. E. 1999 Conditional probabilities, relative operating characteristics, and relative operating levels. Weather and Forecasting, 14, 713–725
  • Mason, S. J., Goddard, L., Graham, N. E., Yulaeva, E., Sun, L. and Arkin, P. A. 1999 The IRI seasonal climate prediction system and the 1997/98 El Niño event. Bull. Am. Meteorol. Soc., 80, 1853–1873
  • Metz, C. E. 1978 Basic principles of ROC analysis. Semin. Nucl. Med., 8, 283–298
  • Metz, C. E. and Kronman, H. B. 1980 Statistical significance tests for binormal ROC curves. J. Math. Psychol., 22, 218–243
  • Metz, C. E., Wang, P. -L. and Kronman, H. B. 1984 A new approach for testing the significance of differences between ROC curves measured from correlated data. In Information processing in medical imaging. Ed. F. Deconinck. Nijhof, The Hague, the Netherlands
  • Mullen, S. L. and Buizza, R. 2001 Quantitative precipitation forecasts over the United States by the ECMWF ensemble prediction system. Mon. Weather Rev., 129, 638–663
  • Murphy, A. H. and Winkler, R. L. 1987 A general framework for forecast verification. Mon. Weather Rev., 115, 1330–1338
  • Nelson, T. O. 1984 A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychol. Bull., 95, 109–133
  • 1986 ROC curves and measures of discrimination accuracy: a reply to Swets. Psychol. Bull., 100, 128–132
  • Neumann, N. 1988 Some procedures for calculating the distributions of nonparametric test statistics. Stat. Software Newsl., 14, 120–126
  • Neyman J. and Pearson, E. S. 1933 On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. London, A231, 289–337
  • Odeh, R. E. 1972 Algorithm AS 55. The generalized Mann–Whitney U-statistic. Appl. Stat., 21, 348–351
  • Palmer, T. N., Branković, and Richardson, D. S. 2000 A probability and decision-model analysis of PROVOST seasonal multi-model ensemble integrations. Q. J. R. Meteorol. Soc., 126, 2013–2033
  • Peterson, W. W. and Birdsall, T. G. 1953 ‘The theory of signal detectability: Part I. The general theory’. Electronic Defense Group, Technical Report 13, June 1953. Available from EECS Systems Office, University of Michigan, 1301 Beal Avenue, Ann Arbor, MI 48109-2122 USA
  • Peterson, W. W., Birdsall, T. G. and Fox, W. C. 1954 The theory of signal detectability. Trans. IRE Prof. Group Inf. Theory, PGIT, 2–4, 171–212
  • Preisendorfer, R. 1988 Principal component analysis in meteorology and oceanography. Elsevier, New York, USA
  • Richardson, D. S. 2000 Skill and relative economic value of the ECMWF ensemble prediction system. Q. J. R. Meteorol. Soc., 126, 649–667
    Direct Link:
  • Sheskin, D. J. 2000 Handbook of parametric and nonparametric statistical procedures. Chapman and Hall, Boca Raton, Florida, USA
  • Shewhart, W. 1931 Economic control of quality of manufactured products. D. van Norstand, New York, USA
  • Simpson, A. J. and Fitter, M. J. 1973 What is the best index of detectability? Psychol. Bull., 80, 481–488
  • Sokal, R. R. and Rohlf, F. J. 1973 Introduction to biostatistics. Freeman, San Francisco, California, USA
  • Stanski, H. R., Wilson, L. J. and Burrows, W. R. 1989 ‘Survey of common verification methods in meteorology’. Research Report No. 89–5, Atmospheric Environment Service, Forecast Research Division, 4905 Dufferin Street, Downsview, Ontario, Canada
  • Swaving, M., Van Houwelingen, H., Ottes, F. P. and Steerneman, T. 1996 Statistical comparison of ROC curves from multiple readers. Med. Decision Making, 16, 143–153
  • Swets, J. A. 1973 The relative operating characteristic in psychology. Science, 182, 990–1000
  • 1979 ROC analysis applied to the evaluation of medical imaging techniques. Invest. Radiol., 14, 109–121
  • Swets, J. A. 1986 Indices of discrimination of diagnostic accuracy: their ROCs and implied models. Psychol. Bull., 99, 100–117
  • 1988 Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293
  • 1995 Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum Associates, Mahwah, New Jersey, USA
  • Swets, J. A. and Birdsall, T. G. 1967 Deferred decision in human signal detection: a preliminary experiment. Perception and Psychophysics, 2, 15–28
  • Swets, J. A. and Pickett, R. M. 1982 Evaluation of diagnostic systems: methods from signal detection theory. Academic Press, New York, USA
  • Swets, J. A., Tanner, W. P. and Birdsall, T. G. 1961 Decision processes in perception. Psychol. Rev., 68, 301–340
  • Swets, J. A., Dawes, R. M. and Monahan, J. 2000 Better decisions through science. Sci. Am., 283, (4), 70–75
  • Thorncroft, C. and Pytharoulis, I. 2001 A dynamical approach to seasonal prediction of Atlantic tropical cyclone activity. Weather and Forecasting, 16, 725–734
  • von Storch, H. and Zwiers, F. W. 1999 Statistical analysis in climate research. Cambridge University Press, Cambridge, UK
  • Wandishin, M. S., Mullen, S. L., Stensrud, D. J. and Brooks, H. E. 2001 Evaluation of a short-range multimodel ensemble system. Mon. Weather Rev., 129, 729–747
  • Wilks, D. S. 1995 Statistical methods in the atmospheric sciences. Academic Press, San Diego, California, USA
  • 2001 A skill score based on economic value for probability forecasts. Meteorol. Appl., 8, 209–219
    Direct Link:
  • Wilson, L. J. 2000 Comments on ‘Probabilistic predictions of precipitation using the ECMWF ensemble prediction system’. Weather and Forecasting, 15, 361–364
  • Winston, H. A. 1988 A comparison of three radar-based severe-storm-detection algorithms on Colorado high plains thunderstorms. Weather and Forecasting, 3, 131–140
  • WMO 2000 Standardized verification system (SVS) for long-range forecasts (LRF). World Meteorological Organization, Geneva, Switzerlandhttp://www.wmo.ch/web/www/DPS/SVS-for-LRF.html
  • Zhang, H. and Casey, T. 2000 Verification of categorical probability forecasts. Weather and Forecasting, 15, 80–89
  • Zhu, Y., Toth, Z., Wobus, R., Richardson, D. and Mylne, K. 2002 The economic value of ensemble-based forecasts. Bull. Am. Meteorol. Soc., 83, 73–83
Get PDF (356K)