Ionic liquid viscosity is one of the most important properties to consider for practical applications. Yet, the connection between local structure and viscosity remains an open question. This article explores the structural origin of differences in the viscosity and viscoelastic relaxation across several ionic liquids, including cations with alkyl, ether, and thioether tails, of the imidazolium and pyrrolidinium families coupled with the NTf2- anion. In all cases, for the systems studied here, we find that pyrrolidinium-based ions are "harder" than their imidazolium-based counterparts. We make a connection between the chemical concept of hardness vs softness and specific structural and structural dynamic quantities that can be derived from scattering experiments and simulations.