Plasmons are usually described in terms of macroscopic quantities such as electric fields and currents. However, as fundamental excitations of metals, they are also quantum objects with internal structure. We demonstrate that this can induce an intrinsic dipole moment which is tied to the quantum geometry of the Hilbert space of plasmon states. This quantum geometric dipole offers a unique handle for manipulation of plasmon dynamics via density modulations and electric fields. As a concrete example, we demonstrate that scattering of plasmons with a nonvanishing quantum geometric dipole from impurities is nonreciprocal, skewing in different directions in a valley-dependent fashion. This internal structure can be used to control plasmon trajectories in two dimensional materials.