In organic solar cells based on polymer:fullerene blends, energy is lost due to electron transfer from polymer to fullerene. Minimizing the difference between the energy of the polymer exciton (ED*) and the energy of the charge transfer state (ECT) will optimize the open-circuit voltage (Voc). In this work, this energy loss ED*-ECT is measured directly via Fourier-transform photocurrent spectroscopy and electroluminescence measurements. Polymer:fullerene photovoltaic devices comprising two different isoindigo containing polymers: P3TI and PTI-1, are studied. Even though the chemical structures and the optical gaps of P3TI and PTI-1 are similar (1.4 eV–1.5 eV), the optimized photovoltaic devices show large differences in Voc and internal quantum efficiency (IQE). For P3TI:PC71BM blends a ED*-ECT of ∼ 0.1 eV, a Voc of 0.7 V and an IQE of 87% are found. For PTI-1:PC61BM blends an absence of sub-gap charge transfer absorption and emission bands is found, indicating almost no energy loss in the electron transfer step. Hence a higher Voc of 0.92 V, but low IQE of 45% is obtained. Morphological studies and field dependent photoluminescence quenching indicate that the lower IQE for the PTI-1 system is not due to a too coarse morphology, but is related to interfacial energetics. Losses between ECT and qVoc due to radiative and non-radiative recombination are quantified for both material systems, indicating that for the PTI-1:PC61BM material system, Voc can only be increased by decreasing the non-radiative recombination pathways. This work demonstrates the possibility of obtaining modestly high IQE values for material systems with a small energy offset (<0.1 eV) and a high Voc.