Although women and men differ in psychological and endocrine stress responses as well as in the prevalence rates of stress-related disorders, knowledge on sex differences regarding stress regulation in the brain is scarce. Therefore, we performed an in-depth analysis of data from 67 healthy participants (31 women, taking oral contraceptives), who were exposed to the ScanSTRESS paradigm in a functional magnetic resonance imaging study. Changes in cortisol, affect, heart rate and neural activation in response to psychosocial stress were examined in women and men as well as potential sex-specific interactions between stress response domains. Stress exposure led to significant cortisol increases, with men exhibiting higher levels than women. Depending on sex, cortisol elevations were differently associated with stress-related responses in striato-limbic structures: higher increases were associated with activations in men but with deactivations in women. Regarding affect or heart rate responses, no sex differences emerged. Although women and men differ in their overall stress reactivity, our findings do not support the idea of distinct neural networks as the base of this difference. Instead, we found differential stress reactions for women and men in identical structures. We propose considering quantitative predictors such as sex-specific cortisol increases when exploring neural response differences of women and men.