Immune responses from prior SARS-CoV-2 infection and COVID-19 vaccination do not prevent re-infections and may not protect against future novel coronaviruses (CoVs). We examined the incidence of and immune differences against human endemic CoVs (eCoV) as a proxy for response against future emerging CoVs. Assessment was among those with known SARS-CoV-2 infection, COVID-19 vaccination but no documented SARS-CoV-2 infection, or neither exposure. Retrospective cohort analyses suggest that prior SARS-CoV-2 infection, but not COVID-19 vaccination alone, protects against subsequent symptomatic eCoV infection. CD8+ T cell responses to the non-structural eCoV proteins, nsp12 and nsp13, were significantly higher in individuals with previous SARS-CoV-2 infection as compared to the other groups. The three groups had similar cellular responses against the eCoV spike and nucleocapsid, and those with prior spike exposure had lower eCoV-directed neutralizing antibodies. Incorporation of non-structural viral antigens in a future pan-CoV vaccine may improve protection against future heterologous CoV infections.