Automatic Item Generation: A More Efficient Process for Developing Mathematics Achievement Items?

Abstract

The continual supply of new items is crucial to maintaining quality for many tests. Automatic item generation (AIG) has the potential to rapidly increase the number of items that are available. However, the efficiency of AIG will be mitigated if the generated items must be submitted to traditional, time‐consuming review processes. In two studies, generated mathematics achievement items were subjected to multiple stages of qualitative review for measuring the intended skills, followed by empirical tryout in operational testing. High rates of success were found. Further, items generated from the same item structure had predictable psychometric properties. Thus, the feasibility of a more limited and expedient review processes was supported. Additionally, positive results were obtained on measuring the same skills from item structures with reduced cognitive complexity.