In heterogeneous landscapes, the genetic and demographic consequences of dispersal influence the evolution of niche width. Unless pollen is limiting, pollen dispersal does not contribute directly to population growth. However, by disrupting local adaptation, it indirectly affects population dynamics. We compare the effect of pollen versus seed dispersal on the evolution of niche width in heterogeneous habitats, explicitly considering the feedback between maladaptation and demography. We consider two scenarios: the secondary contact of two subpopulations, in distinct, formerly isolated habitats, and the colonization of an empty habitat with dispersal between the new and ancestral habitat. With an analytical model, we identify critical levels of genetic variance leading to niche contraction (secondary contact scenario), or expansion (new habitat scenario). We confront these predictions with simulations where the genetic variance freely evolves. Niche contraction occurs when habitats are very different. It is faster as total gene flow increases or as pollen predominates in overall gene flow. Niche expansion occurs when habitat heterogeneity is not too high. Seed dispersal accelerates it, whereas pollen dispersal tends to retard it. In both scenarios very high seed dispersal leads to extinction. Overall, our results predict a wider niche for species dispersing seeds more than pollen.