Acute infectious diseases are transmitted over networks of social contacts. Epidemic models are used to predict the spread of emergent pathogens and to compare intervention strategies. Many of these models assume equal probability of contact within mixing groups (homes, schools, etc.), but little work has inferred the actual contact network, which may influence epidemic estimates. We develop a penalized likelihood method to infer contact networks within households, which are a key area for disease transmission. Using egocentric surveys of contact behaviour in Belgium, we estimate within-household contact networks for six different age compositions. Our estimates show dependence in contact behaviour and vary substantively by age composition, with fewer contacts in older households. Our results are relevant for epidemic models that are used to make policy recommendations.