Considerable evidence suggests that fathers' absence from home has a negative short- and long-term impact on children's health, psychosocial development, cognition and educational experience. We assessed the impact of father presence during infancy and childhood on children's height-for-age z-score (HAZ) at 5 years old. We conducted secondary data analysis from a 15-year cohort study (Young Lives) focusing on one of four Young Lives countries (Peru, n = 1821). When compared with children who saw their fathers on a daily or weekly basis during infancy and childhood, children who did not see their fathers regularly at either period had significantly lower HAZ scores (−0.23, P = 0.0094) after adjusting for maternal age, wealth and other contextual factors. Results also suggest that children who saw their fathers during childhood (but not infancy) had better HAZ scores than children who saw their fathers in infancy and childhood (0.23 z-score, P = 0.0388). Findings from analyses of resilient children (those who did not see their fathers at either round but whose HAZ > −2) show that a child's chances of not being stunted in spite of paternal absence at 1 and 5 years old were considerably greater if he or she lived in an urban area [odds ratio (OR) = 9.3], was from the wealthiest quintile (OR = 8.7) and lived in a food secure environment (OR = 3.8). Interventions designed to reduce malnutrition must be based on a fuller understanding of how paternal absence puts children at risk of growth failure.