Response time (RT) generally slows with aging, but the contribution of structural brain changes to this slowing is unknown. We used voxel-based morphometry (VBM) to determine gray matter (GM) and white matter (WM) brain volumes in 9 middle-aged adults (38-58 years old) and 9 seniors (66-82 years old). We correlated brain volumes with RT assessed in both a simple visual stimulus-response task and a visual continuous recognition memory task. No GM correlations with simple RT were significant; there was one WM correlation in the right fusiform gyrus. In the memory task, faster RT was correlated (p < .05, corrected) with less GM in the globus pallidus, the parahippocampus, and the thalamus for both groups. Several Brodmann areas (BA) differed between the groups such that in each area, less GM was correlated with slower RTs in the middle-aged group but with faster RTs in the senior group (BAs 19, 37, 46, 9, 8, 6, 13, 10, 41, and 7). The results suggest that individual differences in specific brain structure volumes should be considered as potential moderating factors in cognitive brain imaging studies.