The stability of mass transfer is important in the formation of contact binaries from detached binaries when the primaries of the initially detached binaries fill their Roche lobes. Using Eggleton’s stellar evolution code, we investigate the formation and the short-period limit of contact binaries by considering the effect of the instability of mass transfer. It is found that with decreasing initial primary mass from 0.89 to 0.63 M⊙, the range of the initial mass ratio decreases for detached binaries that experience stable mass transfer and evolve into contact. If the initial primary mass is less than 0.63 M⊙, detached binaries would experience dynamically unstable mass transfer when the primaries of detached binaries fill their Roche lobes. These systems would evolve into a common envelope situation and probably then to a complete merger of two components on a quite short time-scale. This results in a low mass limit at about 0.63 M⊙ for the primary mass of contact binaries, which might be the main reason why the period distribution of contact binaries has a short limit of about 0.22 d. By comparing the theoretical period distribution of contact binaries with the observational data, it is found that the observed contact binaries are above the low mass limit for the primary mass of contact binaries and no observed contact binaries are below this limit. This suggests that the short-period limit of contact binaries can be explained by the instability of the mass transfer that occurs when the primaries of the initially detached binaries fill their Roche lobes.