Genomic selection patterns and hybrid performance influence the chance that crop (trans)genes can spread to wild relatives. We measured fitness(-related) traits in two different field environments employing two different crop–wild crosses of lettuce. We performed quantitative trait loci (QTL) analyses and estimated the fitness distribution of early- and late-generation hybrids. We detected consistent results across field sites and crosses for a fitness QTL at linkage group 7, where a selective advantage was conferred by the wild allele. Two fitness QTL were detected on linkage group 5 and 6, which were unique to one of the crop–wild crosses. Average hybrid fitness was lower than the fitness of the wild parent, but several hybrid lineages outperformed the wild parent, especially in a novel habitat for the wild type. In early-generation hybrids, this may partly be due to heterosis effects, whereas in late-generation hybrids transgressive segregation played a major role. The study of genomic selection patterns can identify crop genomic regions under negative selection across multiple environments and cultivar–wild crosses that might be applicable in transgene mitigation strategies. At the same time, results were cultivar-specific, so that a case-by-case environmental risk assessment is still necessary, decreasing its general applicability.