The fitness of populations adapting to new environments is expected to decline in different environments, but empirical studies often do not lend support for such adaptation costs. We test the idea that the initial fitness of the selected populations in the environment where the cost is estimated is key for interpreting tests of ecological trade-offs. We isolated single clones of the yeast Saccharomyces cerevisiae every ~250 generations from replicate experimental lineages that had been selected during 5000 generations in a glucose-limited environment. We then selected these clones in a galactose-limited environment for ~120 generations. Finally, we estimated single-clone fitness in both environments, before and after selection on galactose. The pleiotropic effects on glucose of selection on galactose evolved from positive to negative as fitness in glucose increased, providing strong support for the importance of initial fitness for determining the sign and magnitude of pleiotropic effects. This demonstrates that the sign of pleiotropic effects for fitness following adaptation to a new environment can change during long-term adaptation to an original environment. We also found no relationship between the size of the fitness changes in galactose and glucose, such that pleiotropic effects in glucose became relatively smaller as the sizes of direct effects on galactose increased.