Imaging studies using region-of-interest morphometry and positron emission tomography have contributed to our understanding of structural and functional abnormalities in borderline personality disorder (BPD); however, both methods have practical limitations to their usefulness for exploratory studies of brain-behavior relationships. We used voxel-based morphometry (VBM) in 34 subjects with BPD and 30 healthy control (HC) subjects to study effects of diagnosis, gender, childhood sexual abuse, depressed mood, impulsivity and aggression on group differences. VBM is a computer-based method for whole brain analysis that combines the advantages of a functional study with a structural method. The BPD subjects, diagnosed with the Diagnostic Interview for Borderline Patients and the International Personality Disorders Examination, were compared with 30 HC subjects, with age and gender covaried. Analyses were repeated separately by gender and, in women, by histories of childhood sexual abuse. Depressed mood, impulsivity, and aggression were covaried in separate analyses. Compared with HC, BPD subjects had significant bilateral reductions in gray matter concentrations in ventral cingulate gyrus and several regions of the medial temporal lobe, including the hippocampus, amygdala, parahippocampal gyrus, and uncus. BPD women (and abused BPD women), but not BPD men, had significant reductions in medial temporal lobe, including the amygdala. BPD men, but not BPD women, showed diminished gray matter concentrations in the anterior cingulate gyrus compared with findings in HC subjects. Covarying for depressed mood rendered group differences non-significant in the ventral cingulate but had little effect on differences in medial temporal cortex. Covarying for aggression (LHA) had relatively little effect on group differences, while covarying for impulsivity, as determined by the Barratt Impulsiveness Scale, rendered all previously noted voxel-level group differences non-significant. Diminished gray matter in the prefrontal cortex and the medial temporal cortex may mediate the dysregulation of impulse and affect in BPD. Group differences varied greatly by gender, levels of depression, and impulsivity. VBM is an efficient method for exploratory study of brain-behavior relationships.