We studied the ability of human subjects to memorize the visual information in computer-generated random block patterns defined either by luminance contrast, by color contrast, or by both. Memory performance declines rapidly with increasing inter-stimulus interval, showing a half-life of approximately 3 s. We further show that memory performance declines with eccentricity approximately as a Gaussian function of position. Memory decay functions did not depend on whether the patterns were defined by luminance or color contrast. Changing both luminance and color components of block patterns in conjunction did not improve performance suggesting a single memory mechanism is used to store luminance and color derived pattern information. Our results further suggest that color identity (hue, saturation) and pattern information extracted from color- or luminance-contrast are stored independently of each other.