Previous research on gender differences in work-to-family conflict, and the latter's linkages with workload and health, has largely ignored the influence of private life context. Here, it is hypothesized that gender differences vary across private life contexts. A multiple-group analysis (SEM) is performed on a representative sample (N = 8,593) of the working population in Flanders (Belgium) to examine gender differences in six family configurations, based on having (or not) a partner (working or not), and having children. Women were found to report higher levels of work-to-family conflict than men, but this difference only holds when both partners are earning (with and without children), and in single-income families with children. Gender differences in the strength of the relationships between conflict and both workload and health were found in the overall sample, but were only reflected in specific subgroups. Our results suggest that private life context should receive attention in future research and policymaking aimed at achieving sustainable careers and caring.