After prolonged exposure to a distorted face with expanded or contracted inner features, a subsequently presented normal face appears distorted toward the opposite direction. This phenomenon, termed as face distortion aftereffect (FDAE), is thought to occur as a result of changes in the mechanisms involved in higher order visual processing. However, the extent to which FDAE is mediated by face-specific configural processing is less known. In the present study, we investigated whether similar aftereffects can be induced by stimuli lacking all the typical characteristics of a human face except for its first-order configural properties. We found a significant FDAE after adaptation to a stimulus consisting of three white dots arranged in a triangular fashion and placed in a gray oval. FDAEs occurred also when the adapting and test stimuli differed in size or when the contrast polarity of the adaptor image was changed. However, the inversion of the adapting image as well as the reduction of its contrast abolished the aftereffect entirely. Taken together, our results suggest that higher-level visual areas, which are involved in the processing of facial configurations, mediate the FDAE. Further, while adaptation seems to be largely invariant to contrast polarity, it appears sensitive to orientation and to lower level manipulations that affect the saliency of the inner features.