Using a simple Gaussian-like Ansatz for the phase distribution of a theory with a complex action, we show how the thimble integration for the average phase factor can be plagued by a strong residual sign problem when the phase of the complex integration measure conspires with the constant phase of the integrand along the thimble. This strong sign problem prohibits the accurate computation of the average phase factor when it becomes exponentially small, and causes a strong sensitivity to the parameters describing the phase distribution.