We study the low-frequency behavior of the ac conductivity σ(ω) of a two-dimensional fermion gas subject to a smooth random potential (RP) or random magnetic field (RMF). We find a nonanalytic ∝|ω| correction to Reσ, which corresponds to a 1/t2 long-time tail in the velocity correlation function. This contribution is induced by return processes neglected in Boltzmann transport theory. The prefactor of this |ω| term is positive and proportional to (d/l)2 for the RP, while it is of opposite sign and proportional to d/l in the weak RMF case, where l is the mean free path and d the disorder correlation length. This nonanalytic correction also exists in the strong RMF regime, when the transport is of a percolating nature. The analytical results are supported and complemented by numerical simulations.