In heterogeneous landscapes, divergent selection can favor the evolution of locally adapted ecotypes, especially when interhabitat gene flow is minimal. However, if habitats differ in size or quality, source-sink dynamics can shape evolutionary trajectories. Upland and bottomland forests of the southeastern USA differ in water table depth, light availability, edaphic conditions, and plant community. We conducted a multiyear reciprocal transplant experiment to test whether Elliott's blueberry (Vaccinium elliottii) is locally adapted to these contrasting environments. Additionally, we exposed seedlings and cuttings to prolonged drought and flooding in the greenhouse to assess fitness responses to abiotic stress. Contrary to predictions of local adaptation, V. elliottii families exhibited significantly higher survivorship and growth in upland than in bottomland forests and under drought than flooded conditions, regardless of habitat of origin. Neutral population differentiation was minimal, suggesting widespread interhabitat migration. Population density, reproductive output, and genetic diversity were all significantly greater in uplands than in bottomlands. These disparities likely result in asymmetric gene flow from uplands to bottomlands. Thus, adaptation to a marginal habitat can be constrained by small populations, limited fitness, and immigration from a benign habitat. Our study highlights the importance of demography and genetic diversity in the evolution of local (mal)adaptation.