Recent evidence on visual neglect suggests that each hemisphere maintains a retinotopically organized representation of the visual world contralateral to the current fixation point and that this representation is based not only on analysis of the current retinal input but, equally importantly, on information retrieved from memory. This idea predicts that unilateral damage to memory systems should produce a lateralized impairment of memory for the retinotopically contralateral visual world. To test this prediction we examined visual recognition memory performance in the left and right visual hemifields of patients who had undergone partial unilateral temporal lobe removals for the relief of epilepsy, either in the left hemisphere (n = 5) or the right (n = 5). The patients were given complex artificial scenes to remember, constructed of independent left and right halves, and were then tested for recognition of the left and the right halves separately. Stimuli were exposed tachistoscopically throughout and fixation was maintained on a central position. Patients made significantly more errors with half-scenes in the hemifield contralateral to their removal than in the ipsilateral hemifield, an increase of 50% in the error rate on average. The effect was seen equally in patients with left and right removals. This finding supports the idea that visual memory retrieval is retinotopically organized.