Detecting Spillover Effects: Design and Analysis of Multilevel Experiments

Authors


  • We thank participants of the St. Louis Area Methods meeting, the Networks in Political Science meeting, the Nuffield Networks meeting, the 2008 annual meeting of the Society of Political Methodology, and the 3rd Annual NYU Experimental Political Science Conference for comments on earlier drafts. Special thanks go to Peter Aronow, who helped with data analysis, and to Mark Grebner and Chris Mann, who helped design and deploy the mailings used here. We are grateful to Delia Bailey, Holger Kern, and Dustin Tingley, who provided helpful comments on early drafts, and to the Yale University Faculty of Arts and Sciences High Performance Computing facility and staff. The Institution for Social and Policy Studies at Yale University provided funding but bears no responsibility for the conclusions we draw. This experiment was approved by the University of Chicago’s Institutional Review Board, proposal H09307. Replication files are available at http://www.home.uchicago.edu/~betsy.

Betsy Sinclair is Assistant Professor of Politics, University of Chicago, 5828 S. UniversityAve., Chicago, IL 60637 (betsy@uchicago.edu). Margaret McConnell is Assistant Professor of Global Health Economics, Harvard University, 9 Bow Street, Cambridge, MA 02138 (mmcconne@hsph.harvard.edu). Donald P. Green is Professor of Political Science, Columbia University, 7th Floor, International Affairs Bldg., 420 W. 118th Street, New York, NY 10027 (dpg2110@columbia.edu).

Abstract

Interpersonal communication presents a methodological challenge and a research opportunity for researchers involved in field experiments. The challenge is that communication among subjects blurs the line between treatment and control conditions. When treatment effects are transmitted from subject to subject, the stable unit treatment value assumption (SUTVA) is violated, and comparison of treatment and control outcomes may provide a biased assessment of the treatment’s causal influence. Social scientists are increasingly interested in the substantive phenomena that lead to SUTVA violations, such as communication in advance of an election. Experimental designs that gauge SUTVA violations provide useful insights into the extent and influence of interpersonal communication. This article illustrates the value of one such design, a multilevel experiment in which treatments are randomly assigned to individuals and varying proportions of their neighbors. After describing the theoretical and statistical underpinnings of this design, we apply it to a large-scale voter-mobilization experiment conducted in Chicago during a special election in 2009 using social-pressure mailings that highlight individual electoral participation. We find some evidence of within-household spillovers but no evidence of spillovers across households. We conclude by discussing how multilevel designs might be employed in other substantive domains, such as the study of deterrence and policy diffusion.

Ancillary