In this study, volumes of the whole brain, hemispheres, and frontal lobes of young and elderly adults were quantified by an automated method. Effects of age, sex, and side on absolute and relative volumes of the brain structures were evaluated. Compared with the young group, elderly participants showed a 15% volume loss in the whole brain and hemispheres, and a 22% volume loss in the frontal lobes. The relative volume of the left hemisphere in the elderly group decreased more than that of the right hemisphere. Elderly men showed significantly greater left hemisphere and left frontal lobe volume losses than did elderly women, indicating that the larger left hemisphere relative volume reduction is largely contributed to by selective atrophy of the left frontal lobe volume in elderly men. These results may reflect age- and sex-related functional deterioration in the left brain.