The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models

Authors


  • Brian Greenhill is an Assistant Professor of Government at Dartmouth College, 223 Silsby Hall, HB 6108, Hanover, NH 03755 (brian.d.greenhill@dartmouth.edu). Michael D. Ward is a Professor of Political Science at Duke University, 408 Perkins Library, Campus Box 90204, Durham, NC 27708 (mw160@duke.edu). Audrey Sacks is a Ph.D. candidate in the Department of Sociology, University of Washington, 211 Savery Hall, Box 353340, Seattle, WA 98195-3340 (sacks@u.washington.edu).

  • An earlier version of this article was presented at the 2009 annual meeting of the American Political Science Association, Toronto, Canada. We appreciate the comments and suggestions we received from John Ahlquist, Hyeran Jo, Francisco Pedraza, and from colleagues associated with the ICEWS project, especially Sean O’Brien, Philippe Loustanau, and Laura Stuart. Colleagues at the 10th Anniversary Conference on Statistics and the Social Sciences, at the University of Washington, Seattle, WA, also provided useful input, especially Andrew Gelman and Steve Fienberg. We also presented a version of these ideas at the 2009 Summer Political Methodology Conference at Yale University and received valuable reactions from a variety of colleagues. Despite all this good advice, we remain responsible for results presented herein. This project was partially supported by the Information Processing Technology Office of the Defense Advanced Research Projects Agency aimed at producing models to provide an Integrated Crisis Early Warning Systems (ICEWS) for decision makers in the U.S. defense community. The holding grant is to the Lockheed Martin Corporation, Contract FA8650-07-C-7749. The anonymous reviewers and editor made exceptionally helpful comments. The views presented in this article are those of the authors, not those of any sponsors. Replication data for this article are available at http://dvn.iq.harvard.edu/dvn/dv/briangreenhill.

Abstract

We present a visual method for assessing the predictive power of models with binary outcomes. This technique allows the analyst to evaluate model fit based upon the models’ ability to consistently match high-probability predictions to actual occurrences of the event of interest, and low-probability predictions to nonoccurrences of the event of interest. Unlike existing methods for assessing predictive power for logit and probit models such as Percent Correctly Predicted statistics, Brier scores, and the ROC plot, our “separation plot” has the advantage of producing a visual display that is informative and easy to explain to a general audience, while also remaining insensitive to the often arbitrary probability thresholds that are used to distinguish between predicted events and nonevents. We demonstrate the effectiveness of this technique in building predictive models in a number of different areas of political research.

Ancillary