We analyse subhaloes in the Via Lactea II (VL2) cosmological simulation to look for correlations among their infall times and z = 0 dynamical properties. We find that the present-day orbital energy is tightly correlated with the time at which subhaloes last entered within the virial radius. This energy–infall correlation provides a means to infer infall times for Milky Way satellite galaxies. Assuming that the Milky Way's assembly can be modelled by VL2, we show that the infall times of some satellites are well constrained given only their Galactocentric positions and line-of-sight velocities. The constraints sharpen for satellites with proper motion measurements. We find that Carina, Ursa Minor and Sculptor were all accreted early, more than 8 Gyr ago. Five other dwarfs, including Sextans and Segue 1, are also probable early accreters, though with larger uncertainties. On the other extreme, Leo T is just falling into the Milky Way for the first time while Leo I fell in ∼2 Gyr ago and is now climbing out of the Milky Way's potential after its first perigalacticon. The energies of several other dwarfs, including Fornax and Hercules, point to intermediate infall times, 2–8 Gyr ago. We compare our infall time estimates to published star formation histories and find hints of a dichotomy between ultrafaint and classical dwarfs. The classical dwarfs appear to have quenched star formation after infall but the ultrafaint dwarfs tend to be quenched long before infall, at least for the cases in which our uncertainties allow us to discern differences. Our analysis suggests that the Large Magellanic Cloud crossed inside the Milky Way virial radius recently, within the last ∼4 billion years.