Visually guided behavior is known to involve temporo-parietal, inferotemporal, and prefrontal cortex and each of these areas appears to contribute to visual working memory. We explored the extent to which chronic lesions in one of these cortical areas affect visually guided oculomotor performance. We also explore whether possible impairments become more pronounced with increasing memory load. With this aim we recorded saccadic eye movements in 19 patients with a chronic focal postsurgical lesion in either temporo-parietal, inferior temporal or prefrontal cortex. Their results are compared to those of 19 age-matched volunteers. The subjects performed three different visual search tasks with increasing memory load: Instructed search, cue-guided search and memory-guided search. In addition, the latter task was performed with a short (1 s) and a long (6 s) delay. All tasks required the subjects to make a saccade to a single target presented together with one or three distractors. The results indicate that patients with inferotemporal lesions make the most task-related errors. Saccadic reaction times (SRTs) were significantly prolonged in patients with temporo-parietal and prefrontal lesions, but were unaffected in the patients with lesions in the inferotemporal cortex. The spatial accuracy of saccades was lowest in patients with temporo-parietal lesions. An increase in memory load led to more errors, to longer reaction times and to lower saccadic precision. However, the effect was similar across the three patient groups and the controls. An error analysis indicated that both patients and controls tended to weight global (luminance contrast and form) features higher than local features (line-segment orientation) when making difficult perceptual decisions.