Theories predict that the evolutionary rates of X-linked regions can differ from those of autosomal regions. The male-biased mutation theory predicts a slower rate of neutral substitution on the X chromosome (slow-X evolution), as the X spends less time in male germlines, where more mutations originate per generation than in female germlines. The fast-X theory, however, predicts a faster rate of adaptive substitution on the X chromosome when newly arising beneficial mutations are, on average, partially recessive (fast-X evolution), as the X enjoys a greater efficacy of positive selection. The slow- and fast-X processes are expected to interact as the degree of male-biased mutation can in turn influence the relative rate of adaptive evolution on the X. Here, we investigate lineage-specific variation in, and the interaction of, slow- and fast-X processes using genomic data from four primates. We find consistent evidence for slow-X evolution in all lineages. In contrast, evidence for fast-X evolution exists in only a subset of lineages. In particular, the marmoset lineage, which shows the strongest evidence of fast-X, exhibits the lowest male mutation bias. We discuss the possible interaction between slow- and fast-X evolution and other factors that influence the degrees of slow- and fast-X evolution.