Variance decomposition analysis is often used to examine the degree to which CEOs influence their companies' performance (the so-called CEO effect). Such studies play an important role in a body of literature that investigates the effect of leadership on organizations. In this paper, I argue that these previous studies have an important underlying flaw. Empirically, these studies wrongly attribute the performance effect of randomness—of chance—to the CEO. I demonstrate how randomness can affect the measured effects in a variance decomposition analysis, and I show that this is especially problematic for the measurement of CEO effects. I demonstrate how this results in a greatly inflated CEO effect and develop an approach to correct for it. Copyright © 2013 John Wiley & Sons, Ltd.