Fish with a transgene for growth hormone grow faster than the wild type and may have an advantage in sexual selection due to their larger size and earlier maturation. The cost in these genetically modified organisms (GMOs) is a lower viability of their offspring. The Trojan gene effect is a hypothesis that predicts that the release of such fish in nature can lead to an invasion by GMOs but ultimately decrease population size to extinction. We modelled GMO invasion with Mendelian inheritance of two alleles in one locus and the resulting mating and population dynamics of wild, GMO and hybrid genotypes. Invasion was attempted over a range of initial densities, representing scenarios from accidental escape to large-scale deliberate introduction of the transgenic genotype. Our results show that invasion strongly depends on hybrid fitness, requiring only a low initial density when GMOs and hybrids are preferred in mating. Preference against hybrids results in an invasion threshold, above which mating between GMOs are sufficiently frequent for invasion to take place. GMO invasion may decrease population size, but contrary to earlier studies on the Trojan gene effect, extinctions do not occur. This is due to the lower viability of GMOs being balanced by the decreased number of competitors reducing the effects of density dependence. The results emphasize the importance of initial density, hybrid fitness and density dependence when considering invasion through hybridization.