Why do memory abilities vary so greatly across individuals and cognitive domains? Although memory functions are highly heritable, what exactly is being genetically transmitted? Here we review evidence for the contribution of both common and partially independent inheritance of distinct aspects of memory function. We begin by discussing the assessment of long-term memory and its underlying neural and molecular basis. We then consider evidence for both specialist and generalist genes underlying individual variability in memory, indicating that carving memory into distinct subcomponents may yield important information regarding its genetic architecture. And finally we review evidence from both complex and single-gene disorders, which provide insight into the molecular mechanisms underlying the genetic basis of human memory function.