Interatomic Coulombic decay (ICD) is a mechanism that allows microscopic objects to rapidly exchange energy. When the two objects are distant, the energy transfer between the donor and acceptor species takes place via the exchange of a virtual photon. On the contrary, recent ab initio calculations have revealed that the presence of a third passive species can significantly enhance the ICD rate at short distances due to the effects of electronic wave function overlap and charge transfer states [Phys. Rev. Lett. 119, 083403 (2017)PRLTAO0031-900710.1103/PhysRevLett.119.083403]. Here, we develop a virtual photon description of three-body ICD, allowing us to investigate retardation and geometrical effects which are out of reach for current ab initio techniques. We show that a passive atom can have a significant influence on the rate of the ICD process at fairly large interatomic distances, due to the scattering of virtual photons off the mediator. Moreover, we demonstrate that in the retarded regime ICD can be substantially enhanced or suppressed depending on the position of the ICD-inactive object, even if the latter is far from both donor and acceptor species.