SEARCH

SEARCH BY CITATION

Abstract

In our recent paper, we illustrated that despite modest levels of inter-rater reliability at the manuscript/journal level, the reliability of career-level assessments in the organizational sciences is very high because the high frequency of evaluations that accumulates into extremely reliable estimates of true scores over time. This is not a subjective opinion, but rather a straightforward technical issue. Many of the objections raised regarding this conclusion by a response to our work did not deal with the reliability of the career assessments, but rather the difficulty associated with reaching productivity standards that they felt were unreasonable. What constitutes “reasonable” expectations is a subjective opinion that varies across institutions and individuals. We argue that academic institutions and job applicants need to have an open and honest conversation that would allow sound decision making on the part of both parties, and we believe that our previous simulations can help frame such conversations. Copyright © 2008 John Wiley & Sons, Ltd.