Re-entrant circuits involving communication between the frontal cortex and other brain areas have been hypothesized to be necessary for maintaining the sustained patterns of neural activity that represent information in working memory, but evidence has so far been indirect. If working memory maintenance indeed depends on such temporally precise and robust long-distance communication, then performance on a delayed recognition task should be highly dependent on the microstructural integrity of white-matter tracts connecting sensory areas with prefrontal cortex. This study explored the effect of variations in white-matter microstructure on working memory performance in two separate groups of participants: patients with multiple sclerosis and age- and sex-matched healthy adults. Functional magnetic resonance imaging was performed to reveal cortical regions involved in spatial and object working memory, which, in turn, were used to define specific frontal to extrastriate white-matter tracts of interest via diffusion tensor tractography. After factoring out variance due to age and the microstructure of a control tract (the corticospinal tract), the number of errors produced in the object working memory task was specifically related to the microstructure of the inferior frontal-occipital fasciculus. This result held for both groups, independently, providing a within-study replication with two different types of white-matter structural variability: multiple sclerosis-related damage and normal variation. The results demonstrate the importance of interactions between specific regions of the prefrontal cortex and sensory cortices for a nonspatial working memory task that preferentially activates those regions.