A wealth of empirical evidence is accumulating on the genetic mediation of brain structure phenotypes. This comes from twin studies that assess heritability and genetic covariance between traits, candidate gene associations, and genome-wide association studies (GWAS) that can identify specific genetic variants. Here we review the major findings from each of these approaches and consider how they inform on the genetic architecture of brain structure. The findings from twin studies show there is a strong genetic influence (heritability) on brain structure, and overlap of genetic effects (pleiotropy) between structures, and between structure and cognition. However, there is also evidence for genetic specificity, with distinct genetic effects across some brain regions. Candidate gene associations show little convergence; most have been under powered to detect effect sizes of the magnitude now expected. GWAS have identified 19 genetic variants for brain structure, though no replicated associations account for more than 1% of the variance. Together these studies are revealing new insights into the genetic architecture of brain morphology. As the scope of inquiry broadens, including measures that capture the complexity of the brain, along with larger samples and new analyses, such as genome-wide common trait analysis (GCTA) and polygenic scores, which combine variant effects for a phenotype, as well as whole-genome sequencing, more genetic variants for brain structure will be identified. Increasingly, large-scale multi-site studies will facilitate this next wave of studies, and promise to enhance our understanding of the etiology of variation in brain morphology, as well as brain disorders.