Ever since Bagehot’s (1873) pioneering work, it is a widely accepted wisdom that in order to alleviate (ex ante) bank moral hazard, a lender of last resort should lend at penalty rates only. In a model in which banks are subject to shocks but can exert effort to affect the likelihood of these shocks, we show that the validity of this argument crucially relies on banks always remaining solvent. The reason is that when banks become insolvent, Bagehot’s prescription dictates to let them fail. Penalty rates charged when banks are illiquid (but solvent) then reduce banks’ incentives to avoid insolvency ex ante and thus increase bank moral hazard. We derive a condition which shows precisely when this effect on ex ante incentives outweighs the traditional one and show that it is fulfilled under plausible scenarios.