The relationship between form and function can have profound effects on evolutionary dynamics and such effects may differ for simple versus complex systems. In particular, functions produced by multiple structural configurations (many-to-one mapping, MTOM) may dampen constituent trade-offs and promote diversification. Unfortunately, we lack information about the genetic architecture of MTOM functional systems. The skulls of teleost fishes contain both simple (lower jaw levers) as well as more complex (jaws modeled as 4-bar linkages) functional systems within the same craniofacial unit. We examined the mapping of form to function and the genetic basis of these systems by identifying quantitative trait loci (QTL) in hybrids of two Lake Malawi cichlid species. Hybrid individuals exhibited novelty (transgressive segregation) in morphological components and function of the simple and complex jaw systems. Functional novelty was proportional to the prevalence of extreme morphologies in the simple levers; by contrast, recombination of parental morphologies produced transgression in the MTOM 4-bar linkage. We found multiple loci of moderate effect and epistasis controlling jaw phenotypes in both the simple and complex systems, with less phenotypic variance explained by QTL for the 4-bar. Genetic linkage between components of the simple and complex systems partly explains phenotypic correlations and may constrain functional evolution.