Current Obstacles in Replicating Risk Assessment Findings: A Systematic Review of Commonly Used Actuarial Instruments

Authors

  • Astrid Rossegger Ph.D.,

    1. Molde University College, Institute of Health Sciences, Molde, Norway
    2. Department of Justice, Psychiatric/Psychological Service, Canton of Zurich, Switzerland
    Search for more papers by this author
  • Juliane Gerth M.Sc.,

    1. Molde University College, Institute of Health Sciences, Molde, Norway
    2. Department of Justice, Psychiatric/Psychological Service, Canton of Zurich, Switzerland
    Search for more papers by this author
  • Katharina Seewald M.Sc.,

    1. Molde University College, Institute of Health Sciences, Molde, Norway
    Search for more papers by this author
  • Frank Urbaniok M.D.,

    1. Molde University College, Institute of Health Sciences, Molde, Norway
    2. Department of Justice, Psychiatric/Psychological Service, Canton of Zurich, Switzerland
    Search for more papers by this author
  • Jay P. Singh Ph.D.,

    1. University of Konstanz, Department of Psychology, Konstanz, Germany
    2. University of South Florida, Department of Mental Health and Policy, Tampa, FL, U.S.A
    Search for more papers by this author
  • Jérôme Endrass Ph.D.

    Corresponding author
    1. Department of Justice, Psychiatric/Psychological Service, Canton of Zurich, Switzerland
    • Molde University College, Institute of Health Sciences, Molde, Norway
    Search for more papers by this author

Correspondence to: Jérôme Endrass, Ph.D., Department of Justice, Psychiatric/Psychological Service, Canton of Zurich, Feldstrasse 42, POB 8090, Zürich, Switzerland. E-mail: jerome.endrass@ji.zh.ch

Abstract

An actuarial risk assessment instrument can be considered valid if independent investigations using novel samples can replicate the findings of the instrument's development study. In order for a study to qualify as a replication, it has to adhere to the methodological protocol of the development study with respect to key design characteristics, as well as ensuring that manual-recommended guidelines of test administration have been followed.

A systematic search was conducted to identify predictive validity studies (N = 84) on three commonly used actuarial instruments: the Violence Risk Appraisal Guide (VRAG), the Sex Offender Risk Appraisal Guide (SORAG), and the Static-99. Sample (sex, age, criminal history) and design (follow-up, attrition, recidivism) characteristics, as well as markers of assessment integrity (scoring reliability, item omissions, prorating procedure), were extracted from 84 studies comprising 108 samples.

None of the replications matched the development study of the instrument they were attempting to cross-validate with respect to key sample and design characteristics. Furthermore none of the replications strictly followed the manual-recommended guidelines for the instruments’ administration.

Additional replication studies that follow the methodological protocols outlined in actuarial instruments’ development studies are needed before claims of generalizability can be made. Copyright © 2013 John Wiley & Sons, Ltd.

Ancillary