The temporal lobe is thought to be abnormal in autism, yet standard volumetric analyses are often unrevealing when age, sex, IQ, and head size are controlled. Quantification of temporal lobe structures were obtained in male subjects with autism and controls, where subjects with head circumference (HC) defined macrocephaly were excluded, so that volume differences were not just related to the higher prevalence of macrocephaly in autism. Various statistical methods were applied to the analysis including a classification and regression tree (CART) method, a non-parametric technique that helps define patterns of relationships that may be meaningful in distinguishing temporal lobe differences between subjects with autism and age and IQ matched controls. Subjects with autism were also compared to a separate control group with reading disorder (RD), with the prediction that the temporal lobe morphometric analysis of the reading disorder controls would be more similar to that of the autism group. The CART method yielded a high specificity in classifying autism subjects from controls based on the relationship between the volume of the left fusiform gyrus (LFG) gray and white matter, the right temporal stem (RTS) and the right inferior temporal gyrus gray matter (RITG-GM). Reading disordered individuals were more similar to subjects with autism. Simple size differences did not distinguish the groups. These findings demonstrate different relationships within temporal lobe structures that distinguish subjects with autism from controls. Results are discussed in terms of pathological connectivity within the temporal lobe as it relates to autism.