We report a systematic study of the efficiency limitations of non-fullerene organic solar cells that exhibit a small energy loss (Eloss) between the polymer donor and the non-fullerene acceptor. To clarify the impact of Eloss on the performance of the solar cells, three thieno[3,4-c]pyrrole-4,6-dione-based conjugated polymers (PTPD3T, PTPD2T, and PTPDBDT) are employed as the electron donor, which all have complementary absorption spectra compared with the ITIC acceptor. The corresponding photovoltaic devices show that low Eloss (0.54 eV) in PTPDBDT:ITIC leads to a high open-circuit voltage (Voc) of 1.05 V, but also to a small quantum efficiency, and in turn photocurrent. The high Voc or small energy loss in the PTPDBDT-based solar cells is a consequence of less non-radiative recombination, whereas the low quantum efficiency is attributed to the unfavorable micro-phase separation, as confirmed by the steady-state and time-resolved photoluminescence experiments, grazing-incidence wide-angle X-ray scattering, and resonant soft X-ray scattering (R-SoXS) measurements. We conclude that to achieve high performance non-fullerene solar cells, it is essential to realize a large Voc with small Eloss while simultaneously maintaining a high quantum efficiency by manipulating the molecular interaction in the bulk-heterojunction.