Vegetation at the aquatic–terrestrial interface can alter landscape features through its growth and interactions with sediment and fluids. Even similar species may impart different effects due to variation in their interactions and feedbacks with the environment. Consequently, replacement of one engineering species by another can cause significant change in the physical environment. Here we investigate the species-specific ecological mechanisms influencing the geomorphology of U.S. Pacific Northwest coastal dunes. Over the last century, this system changed from open, shifting sand dunes with sparse vegetation (including native beach grass, Elymus mollis), to densely vegetated continuous foredune ridges resulting from the introduction and subsequent invasions of two nonnative grass species (Ammophila arenaria and Ammophila breviligulata), each of which is associated with different dune shapes and sediment supply rates along the coast. Here we propose a biophysical feedback responsible for differences in dune shape, and we investigate two, non-mutually exclusive ecological mechanisms for these differences: (1) species differ in their ability to capture sand and (2) species differ in their growth habit in response to sand deposition. To investigate sand capture, we used a moveable bed wind tunnel experiment and found that increasing tiller density increased sand capture efficiency and that, under different experimental densities, the native grass had higher sand capture efficiency compared to the Ammophila congeners. However, the greater densities of nonnative grasses under field conditions suggest that they have greater potential to capture more sand overall. We used a mesocosm experiment to look at plant growth responses to sand deposition and found that, in response to increasing sand supply rates, A. arenaria produced higher-density vertical tillers (characteristic of higher sand capture efficiency), while A. breviligulata and E. mollis responded with lower-density lateral tiller growth (characteristic of lower sand capture efficiency). Combined, these experiments provide evidence for a species-specific effect on coastal dune shape. Understanding how dominant ecosystem engineers, especially nonnative ones, differ in their interactions with abiotic factors is necessary to better parameterize coastal vulnerability models and inform management practices related to both coastal protection ecosystem services and ecosystem restoration.