The influence of magnetic Ni and nonmagnetic Zn impurities on the normal-state pseudogap (PG) in the c-axis optical conductivity of (Sm,Nd)Ba(2){Cu(1-y)(Ni,Zn)(y)}(3)O(7-delta) crystals was studied by spectral ellipsometry. We find that these impurities, which strongly suppress superconductivity, have a profoundly different impact on the PG. Zn gives rise to a gradual and inhomogeneous PG suppression while Ni strongly enhances the PG. Our results challenge theories that relate the PG either to precursor superconductivity or to other phases with exotic order parameters, such as flux phase or d-density wave states, that should be suppressed by potential scattering. The apparent difference between magnetic and nonmagnetic impurities instead points towards an important role of magnetic correlations in the PG state.