Genetic predisposition and brain structural abnormalities have been shown to be involved in the biological underpinnings of anorexia nervosa (AN). Prefrontal brain regions are suggested to contribute through behavioral inhibition mechanisms to body weight. However, it is unknown if and to which extent biological correlates for AN might be present in individuals without clinical AN symptomatology. We therefore investigated the contribution of polygenic load for AN on body weight and prefrontal brain structure in a sample of n = 380 nonclinical individuals. A polygenic score (PGS) reflecting the individual genetic load for the trait of anorexia nervosa was calculated. Structural MRI data were acquired and preprocessed using the cortical parcellation stream of FreeSurfer. We observed a significant PGS × sex interaction effect on body mass index (BMI), which was driven by a negative correlation between PGS and BMI in female participants. Imaging analyses revealed significant interaction effects of sex × PGS on surface area of the lateral orbitofrontal cortex (OFC), the pars orbitalis (PO), the rostral middle frontal gyrus (RMF) and the pars triangularis (PT) of the left frontal cortex. The interaction effects were driven by positive correlations between PGS and prefrontal surface areas in female participants and negative correlations in male participants. We furthermore found sex-specific associations between BMI and left RMF surface area as well as between BMI and left PO and left RMF thickness. Our findings demonstrate a sex-specific association between polygenic load for AN, BMI, and prefrontal brain structure in nonclinical individuals. Hence, this study identifies structural abnormalities associated with polygenic load for AN and BMI in brain regions deeply involved in behavioral inhibition and impulse regulation as candidate brain regions for future research.