In many groups of organisms the location of settling is determined by competition, and fitter individuals tend to settle closer to their natal territory than less fit ones. In this work we study the implications of this phenomenon to the problem of adaptation and speciation on a rugged adaptive landscape. One consequence of fitness-associated dispersal (FAD) is that individuals with high fitness are more likely to experience inbreeding, especially with other fit individuals. Another consequence is that when dispersal is costly, the less fit individuals are more likely to pay the cost. When a rare and advantageous allelic combination appears, FAD may increase its chances to spread in the population. In a two-locus two-alleles model with negative epistasis, we find that FAD significantly shortens the waiting time for an adaptive peak shift in comparison with random dispersal.