Are self-assessments reliable indicators of topic knowledge?
Article first published online: 3 FEB 2011
Copyright © 2010 by American Society for Information Science and Technology
Proceedings of the American Society for Information Science and Technology
Volume 47, Issue 1, pages 1–10, November/December 2010
How to Cite
Cole, M. J., Zhang, X., Liu, J., Liu, C., Belkin, N. J., Bierig, R. and Gwizdka, J. (2010), Are self-assessments reliable indicators of topic knowledge?. Proc. Am. Soc. Info. Sci. Tech., 47: 1–10. doi: 10.1002/meet.14504701285
- Issue published online: 3 FEB 2011
- Article first published online: 3 FEB 2011
Self-assessment of topic/task knowledge is a human metacognitive capacity that impacts information behavior, for example through selection of learning and search strategies. It is often used as a measure in experiments for evaluation of results and those measurements are taken to be generally reliable. We conducted a user study (n=40) to test this by constructing a concept-based topic knowledge representation for each participant and then comparing it with the participant judgment of their topic knowledge elicited with Likert-scale questions. The tasks were in the genomics domain and knowledge representations were constructed from the MeSH thesaurus terms that indexed relevant documents for five topics. The participants rated their familiarity with the topic, the anticipated task difficulty, the amount of learning gained during the task, and made other knowledge-related judgments associated with the task. Although there is considerable variability over individuals, the results provide evidence that these self-assessed topic knowledge measures are correlated in the expected way with the independently-constructed topic knowledge measure. We argue the results provide evidence for the general validity of topic knowledge self-assessment and discuss ways to further explore knowledge self-assessment and its reliability for prediction of individual knowledge levels.