Neurodegenerative diseases frequently affect brain regions important for emotional processing, offering a valuable opportunity to study the effects of brain injury on emotion. The current study examined the neuroanatomical correlates of impaired recognition of emotions in patients with neurodegenerative disease. Performance on recognition of facial expressions, as measured by the Florida Affect Battery, was correlated with regional changes in gray matter tissue content in 50 patients with neurodegenerative disease using voxel-based morphometry. Recognition accuracy in the group was poor for negative emotions (fear, anger and sadness) and good for happiness, consistent with previous studies. For negative emotions, a region in the right lateral inferior temporal gyrus (Brodman's area (BA) 20) extending into the right middle temporal gyrus (BA 21) was correlated with accuracy. This effect appeared to be strongest for sadness, which was also independently correlated with atrophy in the superior temporal gyrus. These data suggest that regions in the right lateral and inferolateral temporal lobe are important for visual processing of negative emotions from faces and that functioning of this right temporal network is most critical for recognition of sad faces.