Characterizing the structural development of the neural speech network in early childhood is important for understanding speech acquisition. To investigate speech in the developing brain, 94 children aged 4-7-years-old at risk for early speech disorder were scanned using diffusion weighted imaging (DWI) magnetic resonance imaging (MRI). Additionally, each child completed the Syllable Repetition Task (SRT), a validated measure of phoneme articulation. The DWI data were modeled using multi-compartment restriction spectrum imaging (RSI) to measure restricted and hindered diffusion properties in both grey and white matter. Consequently, we analyzed the diffusion data using both whole brain analysis, and automated fiber quantification (AFQ) analysis to establish tract profiles for each of six fiber pathways thought to be important for supporting speech development. In the whole brain analysis, we found that SRT performance was associated with restricted diffusion in bilateral inferior frontal gyrus ( pars opercularis ), right pre-supplementary/ supplementary motor area (pre-SMA/SMA), and bilateral cerebellar grey matter ( p < .005). Age moderated these associations in left pars opercularis and frontal aslant tract (FAT). However, in both cases only the cerebellar findings survived a cluster correction. We also found associations between SRT performance and restricted diffusion in cortical association fiber pathways, especially left FAT, and in the cerebellar peduncles. Analyses using automatic fiber quantification (AFQ) highlighted differences in high and low performing children along specific tract profiles, most notably in left but not right FAT. These findings suggest that individual differences in speech performance are reflected in structural gray and white matter differences as measured by restricted and hindered diffusion metrics, and offer important insights into developing brain networks supporting speech in very young children.