The analytical foundations of the mode-matching method, based on the Rayleigh principle, are presented for the problem of an unbounded object. This method is applied to the numerical analysis of an embedded optical waveguide. The Rayleigh principle for an unbounded object is derived through the transition process from periodic structure to an open system. In the mode-matching method, the approximate fields are represented by the superposition of plane waves with band-limited spatial spectra. These fields are matched to the boundary conditions in the least squares sense. The uniform convergency of the sequence of the approximate fields is ensured by the Rayleigh principle. The dispersion relation and the field distribution are calculated, and the propagation characteristics of each guided mode are investigated.