Interactions among conspecifics influence social evolution through two distinct but intimately related paths. First, they provide the opportunity for indirect genetic effects (IGEs), where genes expressed in one individual influence the expression of traits in others. Second, interactions can generate social selection when traits expressed in one individual influence the fitness of others. Here, we present a quantitative genetic model of multivariate trait evolution that integrates the effects of both IGEs and social selection, which have previously been modeled independently. We show that social selection affects evolutionary change whenever the breeding value of one individual covaries with the phenotype of its social partners. This covariance can be created by both relatedness and IGEs, which are shown to have parallel roles in determining evolutionary response. We show that social selection is central to the estimation of inclusive fitness and derive a version of Hamilton's rule showing the symmetrical effects of relatedness and IGEs on the evolution of altruism. We illustrate the utility of our approach using altruism, greenbeards, aggression, and weapons as examples. Our model provides a general predictive equation for the evolution of social phenotypes that encompasses specific cases such as kin selection and reciprocity. The parameters can be measured empirically, and we emphasize the importance of considering both IGEs and social selection, in addition to relatedness, when testing hypotheses about social evolution.