We semiclassically derive the leading off-diagonal correction to the spectral form factor of quantum systems with a chaotic classical counterpart. To this end we present a phase space generalization of a recent approach for uniformly hyperbolic systems (Sieber M and Richter K 2001 Phys. Scr. T 90 128, Sieber M 2002 J. Phys. A: Math. Gen. 35 L613). Our results coincide with corresponding random matrix predictions. Furthermore we study the transition from the Gaussian orthogonal to the Gaussian unitary ensemble.