The photophysics of bulk heterojunctions of a high-performance, low-gap silicon-bridged dithiophene polymer with oleic acid capped PbS quantum dots (QDs) are studied to assess the material potential for light harvesting in the visible- and IR-light ranges. By employing a wide range of nanocrystal sizes, systematic dependences of electron and hole transfer on quantum-dot size are established for the first time on a low-gap polymer–dot system. The studied system exhibits type II band offsets for dot sizes up to ca. 4 nm, whch allow fast hole transfer from the quantum dots to the polymer that competes favorably with the intrinsic QD recombination. Electron transfer from the polymer is also observed although it is less competitive with the fast polymer exciton recombination for most QD sizes studied. The incorporation of a fullerene derivative provides efficient electron-quenching sites that improve interfacial polymer-exciton dissociation in ternary polymer–fullerene–QD blends. The study indicates that programmable band offsets that allow both electron and hole extraction can be produced for efficient light harvesting based on this low-gap polymer-PbS QD composite.