Observational studies have revealed a ‘downsizing’ trend in black hole (BH) growth: the number densities of luminous active galactic nuclei (AGN) peak at higher redshifts than those of faint AGN. This would seem to imply that massive BHs formed before low-mass BHs, in apparent contradiction to hierarchical clustering scenarios. We investigate whether this observed ‘downsizing’ in BH growth is reproduced in a semi-analytic model for the formation and evolution of galaxies and BHs, set within the hierarchical paradigm for structure formation. In this model, BHs evolve from light seeds (∼100 M⊙) and their growth is merger driven. The original Somerville et al. model (baseline model) reproduces the number density of AGN at intermediate redshifts and luminosities, but underproduces luminous AGN at very high redshift (z > 3) and overproduces them at low redshift (z < 1). In addition, the baseline model underproduces low-luminosity AGN at low redshift (z < 1). In order to solve these problems, we consider several modifications to the physical processes in the model: (1) a ‘heavy’ BH seeding scenario; (2) a sub-Eddington accretion rate ceiling that depends on the cold gas fraction and (3) an additional BH accretion mode due to disc instabilities. With these three modifications, the models can explain the observed downsizing, successfully reproduce the bolometric AGN luminosity function and simultaneously reproduce galaxy and BH properties in the local Universe. We also perform a comparison with the observed soft and hard X-ray luminosity functions of AGN, including an empirical correction for torus-level obscuration, and reach similar conclusions. Our best-fitting model suggests a scenario in which disc instabilities are the main driver for moderately luminous Seyfert galaxies at low redshift, while major mergers are the main trigger for luminous AGN.