We present near-infrared long-slit and multislit spectra of low-mass brown dwarf candidates in the Orion nebula cluster. The long-slit data were observed in the H and K bands using NIRI on the Gemini-North Telescope. The multi-object spectroscopic observations were made using IRIS2 on the Anglo-Australian Telescope at H band. We develop a spectral typing scheme based on optically calibrated, near-infrared spectra of young sources in the Taurus and IC 348 star-forming regions with spectral types M3.0 to M9.5. We apply our spectral typing scheme to 52 sources, including previously published UKIRT and GNIRS spectra. 40 objects show strong water absorption with spectral types of M3 to >M9.5. The latest type objects are provisionally classified as early L types. We plot our sources on Hertzsprung–Russell diagrams overlaid with theoretical pre-main-sequence isochrones. The majority of our objects lie close to or above the 1-Myr isochrone, leading to an average cluster age that is <1 Myr. We find 38 sources lie at or below the hydrogen-burning limit (0.075 M⊙). 10 sources potentially have masses below the deuterium-burning limit (0.012 M⊙). We use a Monte Carlo approach to model the observed luminosity function with a variety of cluster age and mass distributions. The lowest χ2 values are produced by an age distribution centred at 1 Myr, with a mass function that declines at substellar masses according to an Mα power law in the range α= 0.3–0.6. We find that truncating the mass function at 0.012 M⊙ produces luminosity functions that are starved of the faintest magnitudes, even when using bimodal age populations that contain 10-Myr-old sources. The results of these Monte Carlo simulations therefore support the existence of a planetary mass population in the ONC.