Research Article

A model for quantitative evaluation of an end-to-end question-answering system

  1. Nina Wacholder1,
  2. Diane Kelly2,
  3. Paul Kantor3,
  4. Robert Rittman3,
  5. Ying Sun3,
  6. Bing Bai3,
  7. Sharon Small4,
  8. Boris Yamrom4,
  9. Tomek Strzalkowski4

Article first published online: 23 APR 2007

DOI: 10.1002/asi.20560

Issue

Journal of the American Society for Information Science and Technology

Journal of the American Society for Information Science and Technology

Volume 58, Issue 8, pages 1082–1099, June 2007

Additional Information

How to Cite

Wacholder, N., Kelly, D., Kantor, P., Rittman, R., Sun, Y., Bai, B., Small, S., Yamrom, B. and Strzalkowski, T. (2007), A model for quantitative evaluation of an end-to-end question-answering system. J. Am. Soc. Inf. Sci., 58: 1082–1099. doi: 10.1002/asi.20560

Author Information

  1. 1

    School of Communication, Information and Library Studies, Rutgers University, New Brunswick, NJ 08901

  2. 2

    School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599

  3. 3

    School of Communication, Information and Library Studies, Rutgers University, New Brunswick, NJ 08901

  4. 4

    Department of Computer Science, University at Albany, Albany, NY 12222

Publication History

  1. Issue published online: 22 MAY 2007
  2. Article first published online: 23 APR 2007
  3. Manuscript Accepted: 17 JUL 2006
  4. Manuscript Revised: 1 FEB 2006
  5. Manuscript Received: 15 MAR 2005

SEARCH

SEARCH BY CITATION

ARTICLE TOOLS

View Full Article (HTML) Get PDF (430K)

Abstract

We describe a procedure for quantitative evaluation of interactive question-answering systems and illustrate it with application to the High-Quality Interactive Question-Answering (HITIQA) system. Our objectives were (a) to design a method to realistically and reliably assess interactive question-answering systems by comparing the quality of reports produced using different systems, (b) to conduct a pilot test of this method, and (c) to perform a formative evaluation of the HITIQA system. Far more important than the specific information gathered from this pilot evaluation is the development of (a) a protocol for evaluating an emerging technology, (b) reusable assessment instruments, and (c) the knowledge gained in conducting the evaluation. We conclude that this method, which uses a surprisingly small number of subjects and does not rely on predetermined relevance judgments, measures the impact of system change on work produced by users. Therefore this method can be used to compare the product of interactive systems that use different underlying technologies.

View Full Article (HTML) Get PDF (430K)