We use the Aquarius simulation series to study the imprint of assembly history on the structure of Galaxy-mass cold dark matter haloes. Our results confirm earlier work regarding the influence of mergers on the mass density profile and the inside-out growth of haloes. The inner regions that contain the visible galaxies are stable since early times and are significantly affected only by major mergers. Particles accreted diffusely or in minor mergers are found predominantly in the outskirts of haloes. Our analysis reveals trends that run counter to current perceptions of hierarchical halo assembly. For example, major mergers (i.e. those with progenitor mass ratios greater than 1:10) contribute little to the total mass growth of a halo, on average less than 20 per cent for our six Aquarius haloes. The bulk is contributed roughly equally by minor mergers and by ‘diffuse’ material which is not resolved into individual objects. This is consistent with modelling based on excursion-set theory which suggests that about half of this diffuse material should not be part of a halo of any scale. The simulations themselves suggest that a significant fraction is not truly diffuse, since it was ejected from earlier haloes by mergers prior to their joining the main system. The Aquarius simulations resolve haloes to much lower mass scales than are expected to retain gas or form stars. Thus, the fraction of diffuse dark matter accreted by haloes represents a lower limit to the fraction of diffuse baryons accreted by galaxies. Our results thus confirm that most of the baryons from which visible galaxies form are accreted diffusely, rather than through mergers, and they suggest that only relatively rare major mergers will affect galaxy structure at later times.