A theory of wave scattering from an inhomogeneous irregular layer has been developed with the radiative transfer method. It is shown that there is a basic difference between the plane and irregular layer formulation. When the inhomogeneities are represented by the Rayleigh scatterers, the plane layer problem can be solved with three Fourier components, whereas the irregular layer requires additional Fourier components until convergence is achieved. In general, smaller numbers of Fourier components are required for thicker layers. Comparisons between the backscattering coefficients of a plane versus an irregular Rayleigh layer indicate that the rough surface boundary causes the level difference between the like and the cross polarized returns to be smaller than the plane case. In addition, polarized returns may be dominated by surface roughness effects near vertical incidence. While the cross polarized return continues to be dominated by volume scattering parameters, the interaction between rough surface and volume scattering can cause significant change in both its angular shape and its level.