The Reliability of Survey Assessments of Characteristics of Medical Clinics

Authors

  • Peter V. Marsden,

    Search for more papers by this author
    • Address correspondence to Peter V. Marsden, Ph.D., Professor, Department of Sociology, Harvard University, 630 William James Hall, 33 Kirkland Street, Cambridge, MA 02138. Bruce E. Landon, M.D., M.B.A., Associate Professor, Keith McInnes, M.S., EQHIV Project Director, Lin Ding, Ph.D., Senior Researcher and Paul D. Cleary, Ph.D., Professor, are with the Department of Health Care Policy, Harvard Medical School, Boston, MA. Bruce E. Landon, M.D., M.B.A., Associate Professor, is also a practicing internist in the Division of General Medicine, Beth Israel Deaconess Medical Center, Boston. Ira B. Wilson, M.D., M.Sc., Associate Professor, is with the Institute for Clinical Research and Health Policy Studies and the Department of Medicine, Tufts-New England Medical Center, Boston, MA. Lisa R. Hirschhorn, M.D., M.P.H., Assistant Clinical Professor of Medicine, is with Harvard Medical School and is Senior Clinical Advisor on HIV/AIDS at JSI Research and Training.

  • Bruce E. Landon,

  • Ira B. Wilson,

  • Keith McInnes,

  • Lisa R. Hirschhorn,

  • Lin Ding,

  • Paul D. Cleary


Abstract

Objective. To assess the reliability of survey measures of organizational characteristics based on reports of single and multiple informants.

Data Source. Survey of 330 informants in 91 medical clinics providing care to HIV-infected persons under Title III of the Ryan White CARE Act.

Study Design. Cross-sectional survey.

Data Collection Methods. Surveys of clinicians and medical directors measured the implementation of quality improvement initiatives, priorities assigned to aspects of HIV care, barriers to providing high-quality HIV care, and quality improvement activities. Reliability of measures was assessed using generalizability coefficients. Components of variance and clinician–director differences were estimated using hierarchical regression models with survey items and informants nested within organizations.

Principal Findings. There is substantial item- and informant-related variability in clinic assessments that results in modest or low clinic-level reliability for many measures. Directors occasionally gave more optimistic assessments of clinics than did clinicians.

Conclusions. For most measures studied, obtaining adequate reliability requires multiple informants. Using multiple-item scales or multiple informants can improve the psychometric performance of measures of organizational characteristics. Studies of such characteristics should report the organizational level reliability of the measures used.

Ancillary