Commentary: Measuring What Matters Most
Article first published online: 14 MAR 2013
© 2013 Milbank Memorial Fund
Volume 91, Issue 1, pages 201–204, March 2013
How to Cite
CLANCY, C. M. and MOY, E. (2013), Commentary: Measuring What Matters Most. Milbank Quarterly, 91: 201–204. doi: 10.1111/milq.12008
- Issue published online: 14 MAR 2013
- Article first published online: 14 MAR 2013
We read with great interest Frank and Haw's (2011) “best practice” guidelines for monitoring socioeconomic inequalities in health status because of its direct relevance to our work on the National Healthcare Disparities Report. The National Healthcare Disparities Report is an annual report mandated by Congress on “prevailing disparities in health care delivery as it relates to racial factors and socioeconomic factors” (AHRQ 2012). Our agency, the Agency for Healthcare Research and Quality (AHRQ), leads the production of the report and its companion report, the National Healthcare Quality Report, which focuses on quality of care. Together, the reports track about 250 measures of health care quality and access to provide a snapshot of health care in the United States and to assess changes in health care quality and disparities over time.
Tracking inequalities over time is a relatively new science, and “best practices” are not well defined. Since the publication of the first National Healthcare Reports in 2003, we have developed methods for analyzing and communicating disparities data. Harper and Lynch's guidance (2005) has been particularly helpful in selecting metrics that illustrate specific contrasts. For example, to compare the size of disparities between two populations at a point in time across measures that have different units (percentages, admission rates, death rates), we use relative differences. To assess whether a specific disparity has changed over time—for example, whether the gap in colorectal cancer screening rates between African Americans and whites has grown larger or smaller—we calculate the slopes of regression lines for each population and test whether the slopes differ.
Comparable guidance to aid the selection of measures for tracking inequalities has generally been absent. The National Quality Forum (2012) recently released a protocol for assessing whether quality measures are “disparities sensitive,” but these criteria focus only on disparities related to race, ethnicity, and language. Moreover, most of the measures endorsed by applying this protocol relate to the structure of health care rather than processes or outcomes. Frank and Haw's work to develop guidelines to assess the appropriateness of measures for tracking socioeconomic inequalities in health outcomes thus begins to fill a critical gap in knowledge.
Overall, we agree with most of Frank and Haw's proposed criteria, although we believe there are opportunities to further improve this guidance for measuring inequalities. We agree that completeness and accuracy of reporting, avoidance of reverse causation, and statistical appropriateness are important epidemiological considerations and that clarity of meaning to nonscientists and lack of ambiguity in the indicator's analysis are critical when communicating with policymakers. We also agree that relevance to known social determinants of health is an important criterion for monitoring socioeconomic inequalities. We believe that this relevance criterion could be generalized to apply to other types of inequalities as well, such as race and ethnicity.
In principle, we agree with a criterion related to sensitivity to intervention. In their article, Frank and Haw provide an example of an outcome that cannot be positively affected by policy (rates of mad cow disease after bovine cannibalism was eliminated), and we agree that it is silly to track an outcome that is invariant. When they apply this criterion to the Scottish data, however, they suggest that measures of mortality and hospitalizations are not sensitive to intervention because inequalities have been slow to change. We find this logic a bit tautological and would lead to systems that monitor progress only where there is progress. We propose that some of the discussed inequalities in mortality and hospitalizations may be sensitive to intervention but that effective interventions have not been implemented nationwide. We propose that the essential aspect of this criterion is the potential for change rather than historical observed change.
We think a significant communication characteristic missing from the list of appraisal criteria is importance. Tracking inequalities in health should include outcomes that are valued by people and important to society because they affect large numbers or are costly or are very undesirable.
Looking ahead, we agree with Frank and Haw that measuring children's developmental health and tracking major life events through linked administrative data are promising new approaches to monitoring socioeconomic inequalities. Work is also needed to develop assessment criteria for other types of measures and for purposes other than monitoring the impact of interventions.
At AHRQ, we work with health care providers to improve processes of care and short-term outcomes (e.g., blood pressure control) that lead to better long-term outcomes (e.g., reduced mortality). Providers typically feel more accountable for process measures, since they can impact them more directly. In contrast, many health outcome measures are more strongly affected by environmental conditions than by providers’ actions. Developing criteria for selecting a short list of the best process measures for reducing disparities would improve the monitoring of providers’ performance and reduce the burden of measurement that they currently experience.
Frank and Haw focused on measurement to monitor the success or failure of interventions to reduce socioeconomic inequalities, but there are other reasons to track outcomes. Policymakers may seek analyses of health outcome data to prioritize interventions or to allocate resources. For example, policymakers might want to target interventions for outcomes with large socioeconomic differences that are growing larger or not changing. Policymakers might also want to move resources from areas with small disparities to areas with larger disparities. These other purposes might require different criteria for weighing outcome measures.
As the measurement enterprise continues its exponential growth, we applaud Frank and Haw's efforts to develop guidance to help us select good measures of health outcomes for monitoring socioeconomic inequalities. We think that with some small modifications, this approach can be applied to other types of measures and to uses other than assessing interventions. Helping policymakers glean the information they most need from an overload of data noise will improve the chances that critical disparities can be recognized, targeted, and eliminated.
- AHRQ (Agency for Healthcare Research and Quality). 2012. National Healthcare Disparities Report 2011. AHRQ Publication no. 12-0006. Rockville, MD.
- 2011. Best Practice Guidelines for Monitoring Socioeconomic Inequalities in Health Status: Lessons from Scotland. The Milbank Quarterly 89(4):658–93. , and .
- 2005. Methods for Measuring Cancer Disparities: Using Data Relevant to Healthy People 2010 Cancer-Related Objectives. NCI Cancer Surveillance Monograph Series, no. 6. NIH Publication no. 05-5777. Bethesda, MD: National Cancer Institute. , and .
- National Quality Forum. 2012. Healthcare Disparities and Cultural Competency Consensus Standards: Technical Report. Washington, DC.