To what extent is party loyalty a liability for incumbent legislators? Past research on legislative voting and elections suggests that voters punish members who are ideologically “out of step” with their districts. In seeking to move beyond the emphasis in the literature on the effects of ideological extremity on legislative vote share, we examine how partisan loyalty can adversely affect legislators' electoral fortunes. Specifically, we estimate the effects of each legislator's party unity—the tendency of a member to vote with his or her party on salient issues that divide the two major parties—on vote margin when running for reelection. Our results suggest that party loyalty on divisive votes can indeed be a liability for incumbent House members. In fact, we find that voters are not punishing elected representatives for being too ideological; they are punishing them for being too partisan.