In this work, we theoretically investigate the impact of the atomic scale lattice imperfections of graphene nanoflakes on their nonlinear response enhanced by the resonance between an incident electromagnetic field and localized plasmon. As a case study, we address the second harmonic generation from graphene plasmonic nanoantennas of different symmetries with missing carbon atom vacancy defects in the honeycomb lattice. Using the many-body time-dependent density matrix approach, we find that one defect in the nanoflake comprising over five thousand carbon atoms can strongly impact the nonlinear hyperpolarizability and override the symmetry constraints. The effect reported here cannot be captured using the relaxation time approximation within the quantum or classical framework. Results obtained in this work have thus important implications for the design of nonlinear graphene devices.