Chaotic genetic patchiness denotes unexpected patterns of genetic differentiation that are observed at a fine scale and are not stable in time. These patterns have been described in marine species with free-living larvae, but are unexpected because they occur at a scale below the dispersal range of pelagic larvae. At the scale where most larvae are immigrants, theory predicts spatially homogeneous, temporally stable genetic variation. Empirical studies have suggested that genetic drift interacts with complex dispersal patterns to create chaotic genetic patchiness. Here we use a coancestry model and individual-based simulations to test this idea. We found that chaotic genetic patterns (qualified by global FST and spatio-temporal variation in FST's between pairs of samples) arise from the combined effects of (1) genetic drift created by the small local effective population sizes of the sessile phase and variance in contribution among breeding groups and (2) collective dispersal of related individuals in the larval phase. Simulations show that patchiness levels qualitatively comparable to empirical results can be produced by a combination of strong variance in reproductive success and mild collective dispersal. These results call for empirical studies of the effective number of breeders producing larval cohorts, and population genetics at the larval stage.