Physical distance is a prominent feature in face-to-face social interactions and allows regulating social encounters. Close interpersonal distance (IPD) increases emotional responses during interaction and has been related to avoidance behavior in social anxiety. However, a systematic investigation of the effects of IPD on subjective experience combined with measures of physiological arousal and behavioral responses during real-time social interaction has been missing. Virtual Reality allows for a controlled manipulation of IPD while maintaining naturalistic social encounters. The present study investigates IPD in social interaction using a novel paradigm in Virtual Reality. Thirty-six participants approached virtual agents and engaged in short interactions. IPD was varied between 3.5 and 1 m by manipulating the distance at which agents reacted to the participant's approach. Closer distances were rated as more arousing, less pleasant, and less natural than longer distances and this effect was significantly modulated by social anxiety scores. Skin conductance responses were also increased at short distances compared to longer distances. Finally, an interaction of IPD and social anxiety was observed for avoidance behavior, measured as participants' backward motion during interaction, with stronger avoidance related to close distances and high values of social anxiety. These results highlight the influence of IPD on experience, physiological response, and behavior during social interaction. The interaction of social anxiety and IPD suggests including the manipulation of IPD in behavioral tests in Virtual Reality as a promising tool for the treatment of social anxiety disorder.