Inferring Expertise in Knowledge and Prediction Ranking Tasks
Version of Record online: 17 JAN 2012
Copyright © 2012, Cognitive Science Society, Inc.
Topics in Cognitive Science
Volume 4, Issue 1, pages 151–163, January 2012
How to Cite
Lee, M. D., Steyvers, M., de Young, M. and Miller, B. (2012), Inferring Expertise in Knowledge and Prediction Ranking Tasks. Topics in Cognitive Science, 4: 151–163. doi: 10.1111/j.1756-8765.2011.01175.x
- Issue online: 17 JAN 2012
- Version of Record online: 17 JAN 2012
- Received 15 September 2011; received in revised form 29 October 2011; accepted 6 November 2011
- Ordering problem;
- Ranking problem;
- Wisdom of crowds;
- Model-based measurement
We apply a cognitive modeling approach to the problem of measuring expertise on rank ordering problems. In these problems, people must order a set of items in terms of a given criterion (e.g., ordering American holidays through the calendar year). Using a cognitive model of behavior on this problem that allows for individual differences in knowledge, we are able to infer people's expertise directly from the rankings they provide. We show that our model-based measure of expertise outperforms self-report measures, taken both before and after completing the ordering of items, in terms of correlation with the actual accuracy of the answers. These results apply to six general knowledge tasks, like ordering American holidays, and two prediction tasks, involving sporting and television competitions. Based on these results, we discuss the potential and limitations of using cognitive models in assessing expertise.