We demonstrate spectroscopic photonic force optical coherence elastography (PF-OCE). Oscillations of microparticles embedded in viscoelastic hydrogels were induced by harmonically modulated optical radiation pressure and measured by phase-sensitive spectral-domain optical coherence tomography. PF-OCE can detect microparticle displacements with pico- to nano-meter sensitivity and millimeter-scale volumetric coverage. With spectroscopic PF-OCE, we quantified viscoelasticity over a broad frequency range from 1 Hz to 7 kHz, revealing rich microstructural dynamics of polymer networks across multiple microrheological regimes. Reconstructed frequency-dependent loss moduli of polyacrylamide hydrogels were observed to follow a general power scaling law G''∼ω0.75, consistent with that of semiflexible polymer networks. Spectroscopic PF-OCE provides an all-optical approach to microrheological studies with high sensitivity and high spatiotemporal resolution, and could be especially beneficial for time-lapse and volumetric mechanical characterization of viscoelastic materials.