Get access

Credible Comparisons Using Interpersonally Incomparable Data: Nonparametric Scales with Anchoring Vignettes

Authors


  • Earlier versions of this article were presented at the Robert Wood Johnson Health Policy Scholars annual meeting, the Society for Political Methodology Summer Conference, the annual meeting of the American Political Science Association, and seminars at the University of Michigan and Stanford University. This work has benefited from the comments of the participants at these presentations. Thanks to Martha Bailey, Elizabeth Bruch, Vincent Hutchings, Gary King, Skip Lupia, Walter Mebane, Gerardo Munck, Maisy Samuelson, Jas Sekhon, and Chris Zorn for helpful comments and conversations. Two anonymous reviewers also made helpful comments that improved the presentation of this work. Thanks to the Robert Wood Johnson Foundation and Stanford’s Institute for Research in the Social Sciences (IRiSS) for research support and to the Center for Political Studies and the Department of Health Management and Policy at the University of Michigan for their hospitality while this work was initially written. The data analysis uses the open-source R library anchors (Wand, King, and Lau 2011), which is available at the Comprehensive R Archive Network. The anchors library also contains the data and replication code for this study. I am solely responsible for any errors or omissions.

  • See for examples Aldrich and McKelvey (1977); Brady (1989, 1990); Rossi, Gilula, and Allenby (2001); King et al. (2004); Javaras and Ripley (2007); King and Wand (2007); and Hopkins and King (2010).

  • Examples of anchoring vignettes can be found in studies of personal health (King et al. 2004; Salomon and Murray 2004), studies of job disabilities (Kapteyn, Smith, and Soest 2007), studies of corruption and state effectiveness (Grzymala-Busse 2007), and many other topics. See also the recent symposium on anchoring vignettes in the JRSS-A (Chevalier and Fielding 2011; Van Soest et al. 2011).

Jonathan Wand is Assistant Professor of Political Science, Encina Hall West, Stanford University, Stanford, CA 94305 (wand@stanford.edu).

Abstract

Comparisons of individuals based on their selections from an ordinal scale traditionally assume that all respondents interpret subjective scale categories in exactly the same way. Anchoring vignettes have been proposed as a method to replace this homogeneity assumption with individual-specific data about how each respondent uses the ordinal scale. However, improving interpersonal comparisons with anchoring vignettes also requires a new set of assumptions. In this article, I derive the assumptions needed to make credible nonparametric comparisons using anchoring vignettes, and propose a new nonparametric scale that does not assume homogeneity among respondents. I also provide methods for evaluating empirically whether a set of anchoring objects can produce credible nonparametric interpersonal comparisons. Two empirical studies illustrate the importance of accounting for differences in the use of ordinal scales by showing how our inferences about interpersonal comparisons may change as a function of the assumptions we accept.

Ancillary