Special Issue Article

You have free access to this content

Evaluating Risk Assessments Using Receiver Operating Characteristic Analysis: Rationale, Advantages, Insights, and Limitations

  1. Douglas Mossman M.D.*

Article first published online: 21 JAN 2013

DOI: 10.1002/bsl.2050

Issue

Behavioral Sciences & the Law

Behavioral Sciences & the Law

Special Issue: Methodological Issues in Measuring and Interpreting the Predictive Validity of Violence Risk Assessments

Volume 31, Issue 1, pages 23–39, January/February 2013

Additional Information

How to Cite

Mossman, D. (2013), Evaluating Risk Assessments Using Receiver Operating Characteristic Analysis: Rationale, Advantages, Insights, and Limitations. Behav. Sci. Law, 31: 23–39. doi: 10.1002/bsl.2050

Author Information

  1. Department of Psychiatry and Behavioral Neuroscience

*Correspondence to: Dr. Douglas Mossman, UC Department of Psychiatry, 260 Stetson Street, Suite 3200, Cincinnati, Ohio 45219-0559. E-mail: douglas.mossman@uc.edu

Publication History

  1. Issue published online: 26 FEB 2013
  2. Article first published online: 21 JAN 2013
  3. Manuscript Accepted: 16 OCT 2012
  4. Manuscript Revised: 14 SEP 2012
  5. Manuscript Received: 12 JUN 2012

SEARCH

SEARCH BY CITATION

ARTICLE TOOLS

View Full Article (HTML) Get PDF (7243K)

REFERENCES

  • Anderson, D., & Hanson, R. K. (2010). Static-99: An actuarial tool to assess risk of sexual and violent recidivism among sexual offenders. In Otto R. K., & Douglas K. S. (Eds.), Handbook of Violence Risk Assessment (pp. 251–267). New York: Routledge/Taylor & Francis Group.
  • Barzman, D., Brackenbury, L., Sonnier, L., Schnell, B., Cassedy, A., Salisbury, Sorter M; & Mossman, D. (2011). Brief Rating of Aggression by Children and Adolescents (BRACHA): Development of a tool to assess risk of inpatients' aggressive behavior. The Journal of the American Academy of Psychiatry and the Law, 39, 170–179.
  • Baxtrom v. Herold. (1966). 383 U.S. 107.
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.
  • Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668–1674.
  • Dolan, M., & Doyle, M. (2000). Violence risk prediction. Clinical and actuarial measures and the role of the Psychopathy Checklist. The British Journal of Psychiatry, 177, 303–311.
  • Dorfman, D. D., & Alf, E., Jr. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data, Journal of Mathematical Psychology, 6, 487–496.
  • Douglas, K. S., & Reeves, K. A. (2010). Historical-Clinical-Risk Management-20 (HCR-20) violence risk assessment scheme: Rationale, application, and empirical overview. In Otto R. K., & Douglas K. S. (Eds.), Handbook of violence risk assessment (pp. 147–185). New York, NY: Routledge/Taylor & Francis.
  • Douglas, K. S., Ogloff, J. R. P., Nicholls, T. L., & Grant, I. (1999). Assessing risk for violence among psychiatric patients: The HCR-20 risk assessment scheme and the Psychopathy Checklist: Screening Version. Journal of Consulting and Clinical Psychology, 67, 917–930.
  • Ennis, B. J., & Litwack, T. R. (1974). Psychiatry and the presumption of expertise: Flipping coins in the courtroom. California Law Review, 62, 693–752.
  • Erdman, H. P., Greist, J. H., Gustafson, D. H., Taves, J. E., & Klein, M. H. (1987). Suicide risk prediction by computer interview: a prospective study. The Journal of Clinical Psychiatry, 48, 464–467.
  • Faust, D., & Ahern, D. C. (2011). Clinical judgment and prediction. In Faust D. (Ed.), Coping with psychiatric and psychological testimony 6th ed. (pp. 147–208). New York: Oxford University Press.
  • Gardner, W., Lidz, C. W., Mulvey, E. P., & Shaw, E. C. (1996). A comparison of actuarial methods for identifying repetitively violent patients with mental illnesses. Law and Human Behavior, 20, 35–48.
  • Goodenough, D. J., Rossmann, K., & Lusted, L. B. (1974). Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology, 110, 89–95.
  • Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293–323.
  • Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19–30.
  • Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.
  • Hanson, R. K., & Morton-Bourgon, K. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: A meta-analysis of 118 prediction studies. Psychological Assessment, 21, 1–21.
  • Hart, S. D., & Logan, C. (2011). Formulation of violence risk using evidence- based assessments: the structured professional judgment approach. In Sturmey P., & McMurran M. (Eds.), Forensic case formulation (pp. 83–106). Chichester, West Sussex, UK: John Wiley & Sons.
  • Kans. Stat. Annot. § 59-29a02. (2009).
  • Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28, 3049–3067.
    Direct Link:
  • McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361–365.
  • McNeil, B. J., Varady, P. D., Burrows, B. A., & Adelstein, S. J. (1975). Measures of clinical efficacy—cost-effectiveness calculations in the diagnosis and treatment of hypertensive renovascular disease. The New England Journal of Medicine, 293, 216–221.
  • Metz, C. E., & Kronman, H. B. (1980). Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology 22, 218–243.
  • Metz, C. E., Herman, B. A., & Shen, J. H. (1998). Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Statistics in Medicine, 17, 1033–53.
    Direct Link:
  • Monahan, J. (1981). The Clinical Prediction of Violent Behavior. Rockville, MD: National Institute of Mental Health.
  • Monahan, J. (1984). The prediction of violent behavior: toward a second generation of theory and policy. The American Journal of Psychiatry, 141, 1–11.
  • Monahan, J. (1997). Clinical and actuarial predictions of violence. In Faigman D., Kaye D., Saks M., & Sanders J. (Eds.), Modern scientific evidence: The law of science and expert testimony (Vol. 1 pp. 300–318). St. Paul, MN: West Group.
  • Monahan, J. (2002). Clinical and actuarial predictions of violence. B. Scientific status. In Faigman D. L., Kaye D. H., Saks M. J., & Sanders J. (Eds.), Science and the law: Social and behavioral sciences issues (pp. 90–112). St. Paul, MN: West Group.
  • Morse, S. J. (1978). Crazy behavior, morals, and science: An analysis of mental health law. California Law Review, 51, 527–654.
  • Mossman, D. (1994a) Assessing predictions of violence: being accurate about accuracy. Journal of Consulting and Clinical Psychology, 62, 783–792.
  • Mossman, D. (1994b). Further comments on portraying the accuracy of violence prediction. Law and Human Behavior, 18, 587–593.
  • Mossman, D. (2000). Assessing the risk of violence – are “accurate” predictions useful? The Journal of the American Academy of Psychiatry and the Law, 28, 272–281.
  • Mossman, D. (2006). Critique of pure risk assessment or, Kant meets Tarasoff. University of Cincinnati Law Review, 75, 523–609.
  • Mossman, D., & Somoza, E. (1989). Maximizing diagnostic information from the dexamethasone suppression test: an approach to criterion selection using receiver operating characteristic analysis. Archives of General Psychiatry, 44, 653–660.
  • Murphy, J. M., Berwick, D. M., Weinstein, M. C., Borus, J. F., Budman, S. H., & Klerman, G. L. (1987). Performance of screening and diagnostic tests. Application of receiver operating characteristic analysis. Archives of General Psychiatry, 44, 550–555.
  • Rice, M. E., & Harris, G. T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting and Clinical Psychology, 63, 737–748.
  • Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen's d, and r. Law and Human Behavior, 29, 615–620.
  • Ruscio, J. (2008). A probability-based measure of effect size: Robustness to base rates and other factors. Psychological Methods, 13, 19–30.
  • Simpson, A. J., & Fitter, M. J. (1973). What is the best index of detectability? Psychology Bulletin, 80, 481–488.
  • Singh, J. P., Grann, M., & Fazel, S. (2011). A comparative study of violence risk assessment tools: a systematic review and metaregression analysis of 68 studies involving 25,980 participants. Clinical Psychology Review, 31, 499–513.
  • Somoza, E., & Mossman, D. (1991). ROC curves and the binormal assumption. The Journal of Neuropsychiatry and Clinical Neurosciences, 3, 436–439.
  • Sreenivasan, S., Weinberger, L. E., Frances, A., & Cusworth-Walker, S. (2010). Alice in actuarial land: through the looking glass of changing Static-99 norms. The Journal of the American Academy of Psychiatry and the Law, 38, 400–406.
  • Steadman, H., Mulvey, E., Monahan, J., Robbins, P., Appelbaum, P., Grisso, T., … Silver, E. (1998). Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods. Archives of General Psychiatry, 55, 393–401.
  • Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Sciences in the Public Interest (Supplement to Psychological Science), 1, 1–26.
    Direct Link:
  • Yang, M., Wong, S. C. P., & Coid, J. (2010). The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136, 740–767.
  • Zweig, M. H., & Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry, 39, 561–577.
View Full Article (HTML) Get PDF (7243K)