ARTICLE
Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan
The proposed methods can be implemented via an R package, “list: Statistical Methods for the Item Count Technique and List Experiment” (Blair and Imai 2011), which is freely available at the Comprehensive R Archive Network (http://cran.r‐project.org/package=list). Replication data are available at http://hdl.handle.net/1902.1/21243. Financial support for the survey from Yale's Institute for Social and Policy Studies Field Experiment Initiative and the Macmillan Center for International and Area Studies is gratefully acknowledged. Additional support from the Air Force Office of Scientific Research (Lyall; Grant FA9550‐09‐1‐0314) and the National Science Foundation (Imai; Grant SES–0849715) is also acknowledged. This research was approved by Yale's Human Subjects Committee under IRB protocol #1006006952. We thank Yuki Shiraito for his methodological advice and Aila Matanock, Justin Phillips, and the seminar participants at Kyusyu University, Princeton University, the University of Sydney and the University of Michigan for helpful comments. The editor and three anonymous reviewers provided useful suggestions.
Abstract
List and endorsement experiments are becoming increasingly popular among social scientists as indirect survey techniques for sensitive questions. When studying issues such as racial prejudice and support for militant groups, these survey methodologies may improve the validity of measurements by reducing nonresponse and social desirability biases. We develop a statistical test and multivariate regression models for comparing and combining the results from list and endorsement experiments. We demonstrate that when carefully designed and analyzed, the two survey experiments can produce substantively similar empirical findings. Such agreement is shown to be possible even when these experiments are applied to one of the most challenging research environments: contemporary Afghanistan. We find that both experiments uncover similar patterns of support for the International Security Assistance Force (ISAF) among Pashtun respondents. Our findings suggest that multiple measurement strategies can enhance the credibility of empirical conclusions. Open‐source software is available for implementing the proposed methods.
Citing Literature
Number of times cited according to CrossRef: 62
- Hye‐Sung Kim, Co‐Ethnic and Neighborhood Ties and Financial Social Capital Formation Among the Urban Poor in Kenya, Social Science Quarterly, 10.1111/ssqu.12778, 101, 2, (744-758), (2020).
- Transborder Determinants of Crime, Conflict, and Violence, Violence without Borders: The Internationalization of Crime and Conflict, 10.1596/978-1-4648-1452-5, (43-87), (2020).
- Candelaria Garay, Brian Palmer-Rubin, Mathias Poertner, Organizational and partisan brokerage of social benefits: Social policy linkages in Mexico, World Development, 10.1016/j.worlddev.2020.105103, 136, (105103), (2020).
- Gulnaz Anjum, Adam Chilton, Zahid Usman, United Nations endorsement and support for human rights: An experiment on women’s rights in Pakistan, Journal of Peace Research, 10.1177/0022343320912839, (002234332091283), (2020).
- Francis Naylor, John O’Loughlin, Who are the ‘Don’t Knows’? Missing Data in Surveys of Post-Soviet Conflict-affected Regions, Europe-Asia Studies, 10.1080/09668136.2020.1808192, (1-21), (2020).
- Mara Redlich Revkin, Competitive Governance and Displacement Decisions Under Rebel Rule: Evidence from the Islamic State in Iraq, Journal of Conflict Resolution, 10.1177/0022002720951864, (002200272095186), (2020).
- John McCauley, Steven Finkel, Michael Neureiter, Christopher Belasco, The grapevine effect in sensitive data collection: examining response patterns in support for violent extremism, Political Science Research and Methods, 10.1017/psrm.2020.34, (1-19), (2020).
- Fernanda Rivera Burle, Mathieu Turgeon, Ação afirmativa e desejabilidade social, Opinião Pública, 10.1590/1807-01912020262283, 26, 2, (283-322), (2020).
- GRAEME BLAIR, ALEXANDER COPPOCK, MARGARET MOOR, When to Worry about Sensitivity Bias: A Social Reference Theory and Evidence from 30 Years of List Experiments, American Political Science Review, 10.1017/S0003055420000374, (1-19), (2020).
- Rodrigo Castro Cornejo, Ulises Beltrán, List Experiments, Political Sophistication, and Vote Buying: Experimental Evidence from Mexico, Journal of Politics in Latin America, 10.1177/1866802X20937713, (1866802X2093771), (2020).
- Luke Condra, Austin L. Wright, Civilians, Control, and Collaboration During Civil Conflict, SSRN Electronic Journal, 10.2139/ssrn.3316958, (2019).
- Geoffrey P. R. Wallace, Condemning or Condoning the Perpetrators? International Humanitarian Law and Attitudes Toward Wartime Violence, Law & Social Inquiry, 10.1017/lsi.2018.22, 44, 1, (192-226), (2019).
- Catherine Nickerson, Anup Menon Nandialath, The impact of religious salience on purchase intentions: evidence from the UAE, Journal of Islamic Marketing, 10.1108/JIMA-01-2019-0005, ahead-of-print, ahead-of-print, (2019).
- Chi-lin Tsai, Statistical analysis of the item-count technique using Stata, The Stata Journal: Promoting communications on statistics and Stata, 10.1177/1536867X19854018, 19, 2, (390-434), (2019).
- Austin Knuppe, Local Partners for Local Problems: When Does Foreign Intervention Build Civilian Support?, SSRN Electronic Journal, 10.2139/ssrn.3410558, (2019).
- Kristin Fabbe, Tolga Sınmazdemir, Syrian Refugees in Turkey and the Politics of Postconflict Reconciliation, Review of Middle East Studies, 10.1017/rms.2018.90, 52, 2, (249-262), (2019).
- Richard Traunmüller, Sara Kijewski, Markus Freitag, The Silent Victims of Sexual Violence during War: Evidence from a List Experiment in Sri Lanka, Journal of Conflict Resolution, 10.1177/0022002719828053, (002200271982805), (2019).
- Graeme Blair, Winston Chou, Kosuke Imai, List Experiments with Measurement Error, Political Analysis, 10.1017/pan.2018.56, (1-26), (2019).
- JASON LYALL, YANG-YANG ZHOU, KOSUKE IMAI, Can Economic Assistance Shape Combatant Support in Wartime? Experimental Evidence from Afghanistan, American Political Science Review, 10.1017/S0003055419000698, (1-18), (2019).
- Yimeng Li, Relaxing the No Liars Assumption in List Experiment Analyses, Political Analysis, 10.1017/pan.2019.7, (1-16), (2019).
- Luke N Condra, Austin L Wright, Civilians, Control, and Collaboration during Civil Conflict, International Studies Quarterly, 10.1093/isq/sqz042, (2019).
- Daniel Muise, Jennifer Pan, Online field experiments, Asian Journal of Communication, 10.1080/01292986.2018.1453850, 29, 3, (217-234), (2018).
- Austin L. Wright, Stephen Stapleton, Andres Uribe, Televising Justice During War, SSRN Electronic Journal, 10.2139/ssrn.3237650, (2018).
- Aila M. Matanock, Paul Staniland, How and Why Armed Groups Participate in Elections, Perspectives on Politics, 10.1017/S1537592718001019, 16, 3, (710-727), (2018).
- Anna O. Pechenkina, Andrew W. Bausch, Kiron K. Skinner, The Pitfalls of List Experiments in Conflict Zones, Civil Wars, 10.1080/13698249.2018.1506647, 20, 3, (408-435), (2018).
- Elisabetta De Cao, Clemens Lutz, Sensitive Survey Questions: Measuring Attitudes Regarding Female Genital Cutting Through a List Experiment, Oxford Bulletin of Economics and Statistics, 10.1111/obes.12228, 80, 5, (871-892), (2018).
- Anna V. Andreenkova, Debra Javeline, Sensitive Questions in Comparative Surveys, Advances in Comparative Survey Methods, 10.1002/9781118884997, (139-160), (2018).
- Aila M. Matanock, Miguel García-Sánchez, Does Counterinsurgent Success Match Social Support? Evidence from a Survey Experiment in Colombia, The Journal of Politics, 10.1086/694915, 80, 3, (800-814), (2018).
- Eric Stollenwerk, Securing Legitimacy? Perceptions of Security and ISAF’s Legitimacy in Northeast Afghanistan, Journal of Intervention and Statebuilding, 10.1080/17502977.2018.1504855, (1-21), (2018).
- Andrew M. Linke, Frank D. W. Witmer, John O’Loughlin, J. Terrence McCabe, Jaroslav Tir, Drought, Local Institutional Contexts, and Support for Violence in Kenya, Journal of Conflict Resolution, 10.1177/0022002717698018, 62, 7, (1544-1578), (2017).
- John S. Ahlquist, List Experiment Design, Non-Strategic Respondent Error, and Item Count Technique Estimators, Political Analysis, 10.1017/pan.2017.31, 26, 1, (34-53), (2017).
- Zack Zimbalist, “Fear-of-the-State Bias” in Survey Data, International Journal of Public Opinion Research, 10.1093/ijpor/edx020, 30, 4, (631-651), (2017).
- Jeffrey A. Friedman, Richard Zeckhauser, Analytic Confidence and Political Decision‐Making: Theoretical Principles and Experimental Evidence From National Security Professionals, Political Psychology, 10.1111/pops.12465, 39, 5, (1069-1087), (2017).
- Jason Lyall, Yang-Yang Zhou, Kosuke Imai, Reducing Insurgent Support among At-Risk Populations: Experimental Evidence from Cash Transfers and Livelihood Training in Afghanistan, SSRN Electronic Journal, 10.2139/ssrn.3026531, (2017).
- Megumi Naoi, Weiyi Shi, Boliang Zhu, 'Yes-Man' Firms: Government Campaigns and Policy Positioning of Businesses in China, SSRN Electronic Journal, 10.2139/ssrn.2925386, (2017).
- Richard Traunmmller, Sara Kijewski, Markus Freitag, The Silent Victims of Wartime Sexual Violence: Evidence from a List Experiment in Sri Lanka, SSRN Electronic Journal, 10.2139/ssrn.2937943, (2017).
- Aila M. Matanock, Paul Staniland, How and Why Armed Groups Participate in Elections, SSRN Electronic Journal, 10.2139/ssrn.3079281, (2017).
- Gregory Eady, The Statistical Analysis of Misreporting on Sensitive Survey Questions, Political Analysis, 10.1017/pan.2017.8, 25, 2, (241-259), (2017).
- Katherine B. Coffman, Lucas C. Coffman, Keith M. Marzilli Ericson, The Size of the LGBT Population and the Magnitude of Antigay Sentiment Are Substantially Underestimated, Management Science, 10.1287/mnsc.2016.2503, 63, 10, (3168-3186), (2017).
- Neil Narang, Jessica A. Stanton, A Strategic Logic of Attacking Aid Workers: Evidence from Violence in Afghanistan, International Studies Quarterly, 10.1093/isq/sqw053, 61, 1, (38-51), (2017).
- Kentaro Hirose, Kosuke Imai, Jason Lyall, Can civilian attitudes predict insurgent violence? Ideology and insurgent tactical choice in civil war, Journal of Peace Research, 10.1177/0022343316675909, 54, 1, (47-63), (2017).
- undefined 박종희, undefined Takeshi Iida, undefined 이정환, Consumption Tax Increase and Government Trust : Assessing Citizen Responses to the 2014 Consumption Tax Increase in Japan, Korean Political Science Review, 10.18854/kpsr.2017.51.3.002, 51, 3, (27-52), (2017).
- Daniel W. Gingerich, Virginia Oliveros, Ana Corbacho, Mauricio Ruiz-Vega, When to Protect? Using the Crosswise Model to Integrate Protected and Direct Responses in Surveys of Sensitive Behavior, Political Analysis, 10.1093/pan/mpv034, 24, 2, (132-156), (2017).
- Kosuke Imai, Bethany Park, Kenneth F. Greene, Using the Predicted Responses from List Experiments as Explanatory Variables in Regression Models, Political Analysis, 10.1093/pan/mpu017, 23, 2, (180-196), (2017).
- María del Mar García Rueda, Pier Francesco Perri, Beatriz Rodríguez Cobo, Advances in estimation by the item sum technique using auxiliary information in complex surveys, AStA Advances in Statistical Analysis, 10.1007/s10182-017-0315-2, (2017).
- Winston Chou, Kosuke Imai, Bryn Rosenfeld, Sensitive Survey Questions with Auxiliary Information, Sociological Methods & Research, 10.1177/0049124117729711, (004912411772971), (2017).
- C. Christine Fair, Rebecca Littman, Neil Malhotra, Jacob N. Shapiro, Relative Poverty, Perceived Violence, and Support for Militant Politics: Evidence from Pakistan, Political Science Research and Methods, 10.1017/psrm.2016.6, 6, 01, (57-81), (2016).
- Christoph Mikulaschek, Jacob N. Shapiro, Lessons on Political Violence from America’s Post–9/11 Wars, Journal of Conflict Resolution, 10.1177/0022002716669808, 62, 1, (174-202), (2016).
- Sebastian Schutte, Violence and Civilian Loyalties, Journal of Conflict Resolution, 10.1177/0022002715626249, 61, 8, (1595-1625), (2016).
- James G. Gimpel, Iris Hui, Inadvertent and intentional partisan residential sorting, The Annals of Regional Science, 10.1007/s00168-016-0802-5, 58, 3, (441-468), (2016).
- Gulnaz Anjum, Adam S. Chilton, Zahid Usman, United Nations Endorsement & Support for Human Rights: An Experiment on Women's Rights in Pakistan, SSRN Electronic Journal, 10.2139/ssrn.2736680, (2016).
- Abdul G. Noury, Biagio Speciale, Social constraints and women's education: Evidence from Afghanistan under radical religious rule, Journal of Comparative Economics, 10.1016/j.jce.2016.07.005, 44, 4, (821-841), (2016).
- Bryn Rosenfeld, Kosuke Imai, Jacob N. Shapiro, An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions, American Journal of Political Science, 10.1111/ajps.12205, 60, 3, (783-802), (2015).
- Timothy Frye, Scott Gehlbach, Kyle Lohse Marquardt, Ora John Reuter, Is Putin's Popularity Real?, SSRN Electronic Journal, 10.2139/ssrn.2692049, (2015).
- Lucas Ronconi, Rodrigo Zarazaga, Labor Exclusion and the Erosion of Citizenship Responsibilities, SSRN Electronic Journal, 10.2139/ssrn.2581396, (2015).
- Lucas Ronconi, Rodrigo Zarazaga S.J., Labor Exclusion and the Erosion of Citizenship Responsibilities, World Development, 10.1016/j.worlddev.2015.06.004, 74, (453-461), (2015).
- Eli Berman, Aila M. Matanock, The Empiricists' Insurgency, Annual Review of Political Science, 10.1146/annurev-polisci-082312-124553, 18, 1, (443-464), (2015).
- Susan D. Hyde, Experiments in International Relations: Lab, Survey, and Field, Annual Review of Political Science, 10.1146/annurev-polisci-020614-094854, 18, 1, (403-424), (2015).
- Ronald Rogowski, The Rise of Experimentation in Political Science, Emerging Trends in the Social and Behavioral Sciences, 10.1002/9781118900772, (1-11), (2015).
- Jason Lyall, Yuki Shiraito, Kosuke Imai, Coethnic Bias and Wartime Informing, The Journal of Politics, 10.1086/681590, 77, 3, (833-848), (2015).
- Kentaro Hirose, Kosuke Imai, Jason Lyall, Can Civilian Attitudes Predict Civil War Violence?, SSRN Electronic Journal, 10.2139/ssrn.2446168, (2014).
- Andrew Beath, Fotini Christia, Ruben Enikolopov, Winning Hearts and Minds through Development: Evidence from a Field Experiment in Afghanistan, SSRN Electronic Journal, 10.2139/ssrn.1809677, (2012).
