Validating Student Score Inferences With Person-Fit Statistic and Verbal Reports: A Person-Fit Study for Cognitive Diagnostic Assessment

Authors


Ying Cui and Mary Roduta Roberts, Centre for Research in Applied Measurement and Evaluation, Department of Educational Psychology, 6-110 Educational Building North, University of Alberta, Edmonton, Alberta, Canada, T6G2G5; yc@ualberta.ca.

Abstract

The goal of this study was to investigate the usefulness of person-fit analysis in validating student score inferences in a cognitive diagnostic assessment. In this study, a two-stage procedure was used to evaluate person fit for a diagnostic test in the domain of statistical hypothesis testing. In the first stage, the person-fit statistic, the hierarchy consistency index (HCI; Cui, 2007; Cui & Leighton, 2009), was used to identify the misfitting student item-score vectors. In the second stage, students’ verbal reports were collected to provide additional information about students’ response processes so as to reveal the actual causes of misfits. This two-stage procedure helped to identify the misfits of item-score vectors to the cognitive model used in the design and analysis of the diagnostic test, and to discover the reasons of misfits so that students’ problem-solving strategies were better understood and their performances were interpreted in a more meaningful way.

Ancillary