Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment

Authors


  • The methods proposed in this article as well as other methods useful for designing and analyzing randomized experiments are publicly available as an R package, experiment (Imai 2007), through the Comprehensive R Archive Network (http://cran.r-project.org/). The replication archive is available as Horiuchi, Imai, and Taniguchi (2007). We thank Larry Bartels, Gary Cox, Kentaro Fukumoto, Rachel Gibson, Daniel Ho, Jonathan Katz, Gary King, Matthew McCubbins, James Morrow, Becky Morton, Alison Post, Jas Sekhon, Elizabeth Stuart, and seminar participants at the Australian National University, Harvard University, Princeton University, the University of California, San Diego, and the University of Michigan for their helpful comments. We are also grateful to Takahiko Nagano and Fumi Kawashima of Nikkei Research for administering our experiment and to Jennifer Oh and Teppei Yamamoto for research assistance. We acknowledge financial support from the National Science Foundation (SES-0550873), the Telecommunications Advancement Foundation (Denki Tsūshin Fukyū Zaidan), and the Committee on Research in the Humanities and Social Sciences at Princeton University. Earlier versions of this article were presented at the 2005 annual meeting of the Midwest Political Science Association, the 2005 Summer Political Methodology conference, the 2005 annual meeting of the American Political Science Association, and the 2006 World Congress of the International Political Science Association.

Yusaku Horiuchi is senior lecturer, Crawford School of Economics and Government, the ANU College of Asia and the Pacific, the Australian National University, Canberra, ACT 0200, Australia (yusaku.horiuchi@anu.edu.au). Kosuke Imai is assistant professor of politics, Princeton University, Princeton, NJ 08544-1012 (kimai@Princeton.Edu). Naoko Taniguchi is assistant professor of sociology, Teikyō University, 359 Ōtsuka, Hachiōji, Tōkyō 192-0395, Japan (n-oni@yb3.so-net.ne.jp).

Abstract

Randomized experiments are becoming increasingly common in political science. Despite their well-known advantages over observational studies, randomized experiments are not free from complications. In particular, researchers often cannot force subjects to comply with treatment assignment and to provide the requested information. Furthermore, simple randomization of treatments remains the most commonly used method in the discipline even though more efficient procedures are available. Building on the recent statistical literature, we address these methodological issues by offering general recommendations for designing and analyzing randomized experiments to improve the validity and efficiency of causal inference. We also develop a new statistical methodology to explore causal heterogeneity. The proposed methods are applied to a survey experiment conducted during Japan's 2004 Upper House election, where randomly selected voters were encouraged to obtain policy information from political parties' websites. An R package is publicly available for implementing various methods useful for designing and analyzing randomized experiments.

Ancillary