For cnidarians that can undergo shifts in algal symbiont relative abundance, the underlying algal physiological changes that accompany these shifts are not well known. The sea anemone Anthopleura elegantissima associates with the dinoflagellate Symbiodinium muscatinei and the chlorophyte Elliptochloris marina, symbionts with very different tolerances to light and temperature. We compared the performance of these symbionts in anemones maintained in an 8–11.5 month outdoor common garden experiment with simulated intertidal conditions and three levels of shading (2, 43, and 85% ambient irradiance). Symbiont densities, mitotic indices, photophysiology and pigments were assessed at three time points during the summer, a period of high irradiance and solar heating during aerial exposure. Whereas S. muscatinei was either neutrally or positively affected by higher irradiance treatments, E. marina responded mostly negatively to high irradiance. E. marina in the 85% irradiance treatment exhibited significantly reduced Pmax and chlorophyll early in the summer, but it was not until nearly 3 months later that a shift in symbiont relative abundance toward S. muscatinei occurred, coincident with bleaching. Symbiont densities and proportions remained largely stable in all other treatments over time, and displacement of S. muscatinei by E. marina was not observed in the 2% irradiance treatment despite the potentially better performance of E. marina. While our results support the view that rapid changes in symbiont relative abundance are typically associated with symbiont physiological dysfunction and bleaching, they also show that significant temporal lags may occur between the onset of symbiont stress and shifts in symbiont relative abundances.