Memory difficulties are a frequent cognitive complaint of patients with chronic epilepsy. Previous studies have suggested that the presence of a seizure focus causes reorganization of brain mechanisms underlying memory function. Here we examine whether seizure onset in the left hemisphere and onset in the right hemisphere have different effects on memory lateralization and whether longer duration of epilepsy is associated with increased lateralization of memory functions to the unaffected hemisphere. We hypothesized that hemisphere of onset and duration of epilepsy would influence plasticity of memory mechanisms, similar to the plasticity observed for language mechanisms. Healthy controls (HC, N = 10) and patients with epilepsy (N = 23, 11 with a left- and 12 with a right-hemisphere focus) performed a scene-encoding fMRI task at 4 T. Active voxels (relative to scrambled image viewing) were identified for each participant. Memory laterality indices (LIs) were calculated in three regions of interest (ROIs) designed on the basis of HC group data: a functional ROI, an anatomical-hippocampal ROI, and an anatomical-medial temporal ROI encompassing hippocampus and parahippocampal gyrus. In healthy controls, LIs were suggestive of slight left lateralization of encoding memory for pictures. Patients with right hemisphere epilepsy showed a nonsignificant increase in degree of left lateralization. In contrast, patients with left hemispheric epilepsy showed right-lateralized activation, differing significantly from controls and from patients with right hemispheric epilepsy. Neuropsychological measures of memory (WMS-III Story Recall) across epilepsy patients predicted LIs in the anatomical ROIs: higher scores were associated with more left-lateralized medial temporal fMRI activation. Neither age of onset nor duration of epilepsy was significantly related to LI. These results indicate that focal epilepsy may influence the functional neuroanatomy of memory function.