We address the controversy concerning the necessary conditions for the observation of Berry phases in disordered mesoscopic conductors. For this purpose, we calculate the spin-dependent conductance of disordered two-dimensional structures in the presence of inhomogeneous magnetic fields. Our numerical results show that for both, the overall conductance and quantum corrections, the relevant parameter defining adiabatic spin transport scales with the square root of the number of scattering events, in generalization of Stern's original proposal [A. Stern, Phys. Rev. Lett. 68, 1022 (1992)]. This could hinder a clear-cut experimental observation of Berry phase effects in diffusive metallic rings.