Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles

Authors

  • Ou Lydia Liu,

    Corresponding author
    1. Educational Testing Service
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author
  • Chris Brew,

    Corresponding author
    1. Nuance
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author
  • John Blackmore,

    Corresponding author
    1. Educational Testing Service
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author
  • Libby Gerard,

    Corresponding author
    1. University of California, Berkeley
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author
  • Jacquie Madhok,

    Corresponding author
    1. University of California, Berkeley
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author
  • Marcia C. Linn

    Corresponding author
    1. University of California, Berkeley
    • Ou Lydia Liu, Educational Testing Service, 660 Rosedale Road, MS07-R, Princeton, NJ 08541; lliu@ets.org. Chris Brew, Nuance, 1198 East Arques Avenue, Sunnyvale, CA 94085; cbrew@acm.org. John Blackmore, Educational Testing Service, 660 Rosedale Road, MS75-D, Princeton, NJ 08541; jblackmore@ets.org. Libby Gerard, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; libbygerard@berkeley.edu. Jacquie Madhok, University of California Berkeley, Tolman Hall, Berkeley, CA 94720; jjmadhok@gmail.com. Marcia C. Linn, Graduate School of Education, University of California Berkeley, 4523 Tolman Hall, Berkeley, CA 94720; mclinn@berkeley.edu.

    Search for more papers by this author

Abstract

Content-based automated scoring has been applied in a variety of science domains. However, many prior applications involved simplified scoring rubrics without considering rubrics representing multiple levels of understanding. This study tested a concept-based scoring tool for content-based scoring, c-rater™, for four science items with rubrics aiming to differentiate among multiple levels of understanding. The items showed moderate to good agreement with human scores. The findings suggest that automated scoring has the potential to score constructed-response items with complex scoring rubrics, but in its current design cannot replace human raters. This article discusses sources of disagreement and factors that could potentially improve the accuracy of concept-based automated scoring.

Ancillary