Evaluating rater quality and rating difficulty in online annotation activities
Article first published online: 24 JAN 2013
Copyright © 2012 by American Society for Information Science and Technology
Proceedings of the American Society for Information Science and Technology
Volume 49, Issue 1, pages 1–10, 2012
How to Cite
Organisciak, P., Efron, M., Fenlon, K. and Senseney, M. (2012), Evaluating rater quality and rating difficulty in online annotation activities. Proc. Am. Soc. Info. Sci. Tech., 49: 1–10. doi: 10.1002/meet.14504901166
- Issue published online: 24 JAN 2013
- Article first published online: 24 JAN 2013
Gathering annotations from non-expert online raters is an attractive method for quickly completing large-scale annotation tasks, but the increased possibility of unreliable annotators and diminished work quality remains a cause for concern. In the context of information retrieval, where human-encoded relevance judgments underlie the evaluation of new systems and methods, the ability to quickly and reliably collect trustworthy annotations allows for quicker development and iteration of research.
In the context of paid online workers, this study evaluates indicators of non-expert performance along three lines: temporality, experience, and agreement. It is found that user performance is a key indicator for future performance. Additionally, the time spent by raters familiarizing themselves with a new set of tasks is important for rater quality, as is long-term familiarity with a topic being rated.
These findings may inform large-scale digital collections' use of non-expert raters for performing more purposive and affordable online annotation activities.