There are now large-scale data on which common genetic variants confer risk for attention deficit hyperactivity disorder (ADHD). Here, we use mediation analyses to explore how cognitive and neural features might explain the association between common variant (polygenic) risk for ADHD and its core symptoms. In total, 544 participants participated (mean 21 years, 212 (39%) with ADHD), most with cognitive assessments, neuroanatomic imaging, and imaging of white matter tract microstructure. We found that polygenic risk for ADHD was associated with symptoms of hyperactivity-impulsivity but not inattention. This association was mediated across multiple PRS thresholds by white matter microstructure, specifically by axial diffusivity of the right corona radiata, (maximum indirect effect β = -0.034 (CI: -0.065 to -0.01), by thickness of the left dorsomedial prefrontal (β = -0.029; CI: -0.061 to -0.0047) and area of the right lateral temporal cortex (β = 0.024; CI: 0.0034-0.054). In addition, modest serial mediation was found, mapping a pathway from polygenic risk, to white matter microstructure of the anterior corona radiata, then cognition (working memory, focused attention), and finally to hyperactivity-impulsivity (working memory β = -0.014 (CI: -0.038 to -0.0026); focused attention β = -0.011 (CI: -0.033 to -0.0017). These mediation pathways were diagnostically specific and were not found for polygenic risk for ASD or schizophrenia. In conclusion, using a deeply phenotyped cohort, we delineate a pathway from polygenic risk for ADHD to hyperactive-impulsive symptoms through white matter microstructure, cortical anatomy, and cognition.