Divergence in mating signals is a primary factor leading to reproductive isolation, and thus, speciation. However, the genetic changes underlying such divergence are poorly understood, especially in vertebrates. We used two species of poeciliid fishes, Poecilia velifera and P. mexicana, to explore the link between genes and mating behaviors that has resulted in pre-mating reproductive isolation between these species. Using backcross hybrids created from the F1 male offspring of reciprocal interspecific crosses between a sailfin molly (P. velifera) and a shortfin molly (P. mexicana), we examined the effects of Y-linkage and autosomal contributions on the expression of two male mating behaviors: courtship displays and gonopodial thrusts. The F1 hybrid males displayed a strong influence of sire on courtship display rates, with F1 males sired by the sailfin species showing courtship display rates that were up to three times higher than the rates of displays performed by F1 males sired by the shortfin species. These results suggest a Y-linked genetic effect on the expression of courtship display behavior. Comparisons between backcross hybrid males with sailfin Y-chromosomes or shortfin Y-chromosomes suggested that the interaction of autosomal genes also influences the inheritance of courtship display rates. Sailfin autosomal genes significantly increased the probability of performing courtship displays for hybrid males, and increased display rate for males from the sailfin Y-chromosome line. Autosomal genes had less of an impact on gonopodial thrusting behavior, however, thrust rates did significantly decrease with an increasing proportion of sailfin autosomes in males from the shortfin Y-chromosome line. These results suggest that the inheritance of species differences in mating signals between shortfin and sailfin mollies involves both genes found on the Y-chromosome and autosomal gene influences on their expression.