Based on a Lagrangian description of fluid particle dispersion we suggest that there is a single expression for the vertical eddy diffusivity for all scalars following fluid particles in stably stratified flows. This expression is the same as the Osborn-Cox diffusivity for buoyancy. To test this hypothesis we carry out turbulence simulations with stable background stratification by solving the Boussinesq equations with random forcing together with the equation for a passive scalar with an initial vertical Gauss profile. The development of the mean scalar concentration is studied for three different values of the width of the profile, σ. It is found that the passive scalar diffuses in very good agreement with the classical diffusion equation if the ratio between σ and a turbulent length scale is large enough. The associated eddy diffusivity agrees exactly with the Osborn-Cox diffusivity for buoyancy.