Much is to be learned about the interface between immune responses to SARS-CoV-2 infection and vaccination. We monitored immune responses specific to SARS-CoV-2 Spike Receptor-Binding-Domain (RBD) in convalescent individuals for eight months after infection diagnosis and following vaccination. Over time, neutralizing antibody responses, which are predominantly RBD specific, generally decreased, while RBD-specific memory B cells persisted. RBD-specific antibody and B cell responses to vaccination were more vigorous than those elicited by infection in the same subjects or by vaccination in infection-naïve comparators. Notably, the frequencies of double negative B memory cells, which are dysfunctional and potentially pathogenic, increased in the convalescent subjects over time. Unexpectedly, this effect was reversed by vaccination. Our work identifies a novel aspect of immune dysfunction in mild/moderate COVID-19, supports the practice of offering SARS-CoV-2 vaccination regardless of infection history, and provides a potential mechanistic explanation for the vaccination-induced reduction of "Long-COVID" symptoms.