Solving radiation and scattering problems in the vicinity of a half-space or stratified medium with integral equation methods is complicated, and slowed by computation of expensive Sommerfeld integrals. In this paper, a new formulation of the spectral layered media Green's function is presented. It is demonstrated for a simple scattering problem in the new fast algorithm called the Multipole-Free Fast Inhomogeneous Plane Wave Algorithm (MF-FIPWA). MF-FIPWA is shown to scale as (N log N) in memory usage and processing time. Additional advantages include rigorous treatment of reflected contributions, simplicity in design, and mathematical insight into the fast algorithm.