We consider X-ray binaries (XBs) as potential sources of stellar feedback. XBs observationally appear able to deposit a high fraction of their power output into their local interstellar medium, which may make them a non-negligible source of energy input. The formation rate of the most luminous XBs rises with decreasing metallicity, which should increase their significance during galaxy formation in the early Universe. We also argue that stochastic effects are important to XB feedback (XBF) and may dominate the systematic changes due to metallicity in many cases. Large stochastic variation in the magnitude of XBF at low absolute star formation rates provides a natural reason for diversity in the evolution of dwarf galaxies which were initially almost identical, with several per cent of such haloes experiencing energy input from XBs roughly two orders of magnitude above the most likely value. These probability distributions suggest that the effect of XBF is most commonly significant for total stellar masses between approximately 107 and 108 M⊙, which might resolve a current problem with modelling populations of such galaxies. We explain how XBs might inject energy before luminous supernovae (SNe) contribute significantly to feedback and how XBs can assist in keeping gas hot long after the last core-collapse SN has exploded. Energy input from XBs produces different behaviour to that from SNe, partly since the peak energy input from a mean XB population continues for ≈100 Myr after the start of a starburst. XBF could be especially important to some dwarf galaxies, potentially heating gas without expelling it; the properties of XBF also match those previously derived as allowing episodic star formation. We also argue that the efficiency of SN feedback (SNF) might be reduced when XBF has had the opportunity to act first. In addition, we note that the effect of SNF is unlikely to be scale-free; galaxies smaller than ≈100 pc might well experience less effective SNF.