In this work we investigate the accretion of cosmological fluids on to an intermediate-mass black hole at the centre of a globular cluster, focusing on the influence of the parent stellar system on the accretion flow. We show that the accretion of cosmic background radiation and the so-called dark energy on to an intermediate-mass black hole is negligible. On the other hand, if cold dark matter has a non-vanishing pressure, the accretion of dark matter is large enough to increase the black hole mass well beyond the present observed upper limits. We conclude that either intermediate-mass black holes do not exist, or dark matter does not exist, or it is not strictly collisionless. In the latter case, we set a lower limit for the parameter of the cold dark matter equation of state.