The authors with to acknowledge the helpful comments of two anonymous reviewers, as well as Wei Pan. The opinions expressed in this paper are solely those of the authors. Direct correspondence to Kenneth Frank, Department of Sociology, Michigan State University, 462 Erickson Hall, East Lansing, MI 48824-1034; e-mail: email@example.com.
INDICES OF ROBUSTNESS FOR SAMPLE REPRESENTATION
Article first published online: 25 JUL 2007
Volume 37, Issue 1, pages 349–392, December 2007
How to Cite
Frank, K. and Min, K.-S. (2007), INDICES OF ROBUSTNESS FOR SAMPLE REPRESENTATION. Sociological Methodology, 37: 349–392. doi: 10.1111/j.1467-9531.2007.00186.x
- Issue published online: 25 JUL 2007
- Article first published online: 25 JUL 2007
Social scientists are rarely able to gather data from the full range of contexts to which they hope to generalize (Shadish, Cook, and Campbell 2002). Here we suggest that debates about the generality of causal inferences in the social sciences can be informed by quantifying the conditions necessary to invalidate an inference. We begin by differentiating the target population into two subpopulations: a potentially observed subpopulation from which all of a sample is drawn and a potentially unobserved subpopulation from which no members of the sample are drawn but which is part of the population to which policymakers seek to generalize. We then quantify the robustness of an inference in terms of the conditions necessary to invalidate an inference if cases from the potentially unobserved subpopulation were included in the sample. We apply the indices to inferences regarding the positive effect of small classes on achievement from the Tennessee class size study and then consider the breadth of external validity. We use the statistical test for whether there is a difference in effects between two subpopulations as a baseline to evaluate robustness, and we consider a Bayesian motivation for the indices and compare the use of the indices with other procedures. In the discussion we emphasize the value of quantifying robustness, consider the value of different quantitative thresholds, and conclude by extending a metaphor linking statistical and causal inferences.