The extragalactic background light (EBL), i.e. the diffuse metagalactic photon field in the ultraviolet to infrared, is dominated by the emission from stars in galaxies. It is, therefore, intimately connected with the integrated star formation rate density (SFRD). In this paper, the SFRD is constrained using recent limits on the EBL density derived from observations of distant sources of high and very high energy gamma-rays. The stellar EBL contribution is modelled utilizing simple stellar population spectra including dust attenuation and emission. For modelling the SFRD up to z = 4, a broken power-law function in z + 1 is assumed. A wide range of values for the different model parameters [SFRD(z), metallicity, dust absorption] is investigated, and their impact on the resulting EBL is studied. The calculated EBL densities are compared with the specific EBL density limits, and constraints on the SFRD are derived. For the fiducial model, by adopting a Chabrier initial mass function (IMF) and a second power-law index for the SFRD of β = 0.3, the SFRD is constrained to ≲0.1 and < 0.2 M⊙ yr−1 Mpc−3 for a redshift of z ∼ 1 and z ∼ 2, respectively. The limits for a redshift of z ∼ 1 are in tension with SFRD measurements derived from instantaneous star formation tracers. While the tension for the conservative fiducial model in this study is not yet overly strong, the tension increases when applying plausible changes to the model parameters, e.g., using a Salpeter IMF instead of a Chabrier one or adopting a sub-solar metallicity.