Reading facial emotion is disrupted by both psychopathology, such as autism, and altered function of neurotransmitter, such as serotonin. These effects could result from reduced sensitivity of emotional processing systems to facial emotion. The impact of facial expression is also greater when personally directed than when averted. We therefore hypothesized that brain activity associated with emotional representation, would be more susceptible to manipulation of serotonin function by Acute Tryptophan Depletion (ATD) for front-viewed than side-viewed faces, measured using functional imaging (fMRI). ATD reduced activity independent of face view in left superior temporal sulcus (STS) and anterior cingulate. In temporal pole, medial frontal cortex and orbitofrontal cortex, ATD also reduced activity, but specifically for front-viewed faces. In right STS, ATD increased activity, but specifically for side-viewed faces. Activity in the amygdalae depended on face view and emotion type. We suggest that engagement of empathic and associative learning functions when viewing faces is facilitated by direct facial view and intact serotonin transmission. Averted faces, and reduced serotonin function facilitate attention to the external goal of gaze. These changes could be adaptive in a threatening context and markedly affect empathic function in conditions associated with impaired serotonin function, such as depression and autism.