Strengthening the Experimenter’s Toolbox: Statistical Estimation of Internal Validity

Authors


  • Authors are in alphabetical order. For helpful comments and discussion, we thank Jake Bowers, Sanford Gordon, Kosuke Imai, Cindy Kam, Walter Mebane, David Nickerson, Jasjeet Sekhon, Nicholas Valentino, Lynn Vavrek, and seminar participants at the University of Michigan, Columbia University, and Princeton University. We also thank James Fowler and Cindy Kam for generously sharing their data. For research assistance, we thank Kevin Duska. A previous version of this article was presented at the 2008 annual meeting of the Midwest Political Science Association, the 2008 annual meeting of the American Political Science Association, Boston, MA, and the 2010 Visions in Methodology Conference, Iowa City, IA. Replication materials are available at http://dvn.iq.harvard.edu/dvn/dv/ljk. An online supplement contains further details about specific randomization tests as well as information about the data used.

Luke Keele is Associate Professor, Department of Political Science, 211 Pond Lab, Penn State University, University Park, PA 16802 (ljk20@psu.edu). Corrine McConnaughy is Assistant Professor, Department of Political Science, 2018 Derby Hall, Ohio State University, Columbus, OH 43210 (mcconnaughy.3@polisci.osu.edu). Ismail White is Assistant Professor, Department of Political Science, 2008 Derby Hall, Ohio State University, Columbus, OH 43210 (white.697@osu.edu).

Abstract

Experiments have become an increasingly common tool for political science researchers over the last decade, particularly laboratory experiments performed on small convenience samples. We argue that the standard normal theory statistical paradigm used in political science fails to meet the needs of these experimenters and outline an alternative approach to statistical inference based on randomization of the treatment. The randomization inference approach not only provides direct estimation of the experimenter’s quantity of interest—the certainty of the causal inference about the observed units—but also helps to deal with other challenges of small samples. We offer an introduction to the logic of randomization inference, a brief overview of its technical details, and guidance for political science experimenters about making analytic choices within the randomization inference framework. Finally, we reanalyze data from two political science experiments using randomization tests to illustrate the inferential differences that choosing a randomization inference approach can make.

Ancillary