Classically, the CNS is described as displaying immune privilege, as it shows attenuated responses to challenge by alloantigen. However, the CNS does show local inflammation in response to infection. Although pathogen access to the brain parenchyma and retina is generally restricted by physiological and immunological barriers, certain pathogens may breach these barriers. In the CNS, such pathogens may either cause devastating inflammation or benefit from immune privilege in the CNS, where they are largely protected from the peripheral immune system. Thus, some pathogens can persist as latent infections and later be reactivated. We review the consequences of immune privilege in the context of CNS infections and ask whether immune privilege may provide protection for certain pathogens and promote their latency.