We found a binary active site of surface enhanced Raman scattering (SERS) and surface enhanced hyper Raman scattering (SEHRS) in single Ag nanoaggregates by single particle spectroscopy. The intensity fluctuation of SEHRS was canceled by dividing SEHRS intensity by SERS intensity on the basis of the binary active site analysis. Thanks to the identification of the plasmons common to both SERS and SEHRS we revealed that an enormous enhancement in SEHRS is ascribed to the electromagnetic (EM) enhancement entirely coupled with longitudinal plasmon modes excluding other kinds of enhancement factors. Our results indicate that EM enhancement factors of metal nanostructures are estimated from spectral information on the longitudinal plasmon resonance band obtained by the scattering or absorption spectra of the nanostructures.