Measuring sediment flux in rivers remains a significant problem in studies of landscape evolution. Recent studies suggest that observations of seismic noise near rivers can help provide such measurements, but the lack of models linking observed seismic quantities to sediment flux has prevented the method from being used. Here, we develop a forward model to describe the seismic noise induced by the transport of sediment in rivers. The model provides an expression for the power spectral density (PSD) of the Rayleigh waves generated by impulsive impacts from saltating particles which scales linearly with the number of particles of a given size and the square of the linear momentum. After incorporating expressions for the impact velocity and rate of impacts for fluvially transported sediment, we observe that the seismic noise PSD is strongly dependent on the sediment size, such that good constraints on grain size distribution are needed for reliable estimates of sediment flux based on seismic noise observations. The model predictions for the PSD are consistent with recent measurements and, based on these data, a first attempt at inverting seismic noise for the sediment flux is provided.