Effects of method and format on subjects' responses to a control of variables reasoning problem



Excessive time and training demands have rendered Piaget's clinical method of reasoning assessment impractical for researchers and science teachers who work with large numbers of students. The published literature indicates that reliable, valid alternatives to clinical assessment are feasible. However, the overestimate/underestimate of reasoning for different methods and formats remains unresolved through research. The objective of this study was to determine the effects of various methods and formats on subjects' responses to a Piagetian reasoning problem requiring control of variables. The task chosen for this investigation was the Mealworm problem. The task was presented by three methods: (1) individual clinical interview; (2) group presentation of task followed by paper-and-pencil problem with illustration; and (3) group administration of paper-and-pencil instrument with illustration. Each method included four formats: (1) completion answer with essay justification; (2) completion answer with multiplechoice justification; (3) multiple-choice answer with essay justification; and (4) multiple-choice answer with multiple-choice justification. Two hundred and fifty-three (253) students who were enrolled in a freshman level biological science class participated in the study. The research design was a 3 × 4 factorial design with method and format of assessment as the main effects. The participants were in 12 randomly selected laboratory or discussion sections, and each section was randomly assigned to a cell in the research design. Regression analysis with the individual as the unit of analysis showed that format but not method of assessment accounted for a significant amount of variance in student performance. The overall interaction was nonsignificant. Regression analysis with sections as the unit of analysis revealed nonsignificant findings for both method and format. The interrater agreement in the scoring of the Mealworm task was 82.1%. The principal conclusion is that format but not method of task administration influenced the performance of subjects, and the influence is similar for various combinations of method and format. Discussion is focused on the relative importance of the nonsignificant and significant findings for using alternative methods of Piagetian assessment. The results are also discussed with respect to earlier work, and suggestions for future research as well as implications for teachers are set forth.