The negative impact of external interference on working memory (WM) performance is well documented; yet, the mechanisms underlying this disruption are not sufficiently understood. In this study, electroencephalogram and functional magnetic resonance imaging (fMRI) data were recorded in separate experiments that each introduced different types of visual interference during a period of WM maintenance: distraction (irrelevant stimuli) and interruption (stimuli that required attention). The data converged to reveal that regardless of the type of interference, the magnitude of processing interfering stimuli in the visual cortex (as rapidly as 100 ms) predicted subsequent WM recognition accuracy for stored items. fMRI connectivity analyses suggested that in the presence of distraction, encoded items were maintained throughout the delay period via connectivity between the middle frontal gyrus and visual association cortex, whereas memoranda were not maintained when subjects were interrupted but rather reactivated in the postinterruption period. These results elucidate the mechanisms of external interference on WM performance and highlight similarities and differences of distraction and multitasking.