The cutoff condition of HE, EH, TE, and TM modes is formulated in an integral equation by using a Sturm-Liouville-type boundary problem. A formula for calculating the cutoff frequencies of the TM01 and HE21 modes is derived in addition to the TE01 cutoff by calculating the eigenvalue of this integral equation. The cutoff frequencies of the TM01 and HE21 modes are calculated and compared with those of the TE01 mode for various index profiles. As a result, the cutoff frequency of the TM01 mode is dominant for the single-mode limit over frequencies of the TE01, TM01, and HE21 modes for the α-power law index profile. However, in the case of an index profile including a center dip, the cutoff frequency of the TE01 mode dominates the single-mode condition.