We present a semiclassical calculation of the generalized form factor K-ab(tau) which characterizes the fluctuations of matrix elements of the operators (a) over cap and (b) over cap in the eigenbasis of the Hamiltonian of a chaotic system. Our approach is based on some recently developed techniques for the spectral form factor of systems with hyperbolic and ergodic underlying classical dynamics and f=2 degrees of freedom, that allow us to go beyond the diagonal approximation. First we extend these techniques to systems with f>2. Then we use these results to calculate K-ab(tau). We show that the dependence on the rescaled time tau (time in units of the Heisenberg time) is universal for both the spectral and the generalized form factor. Furthermore, we derive a relation between K-ab(tau) and the classical time-correlation function of the Weyl symbols of (a) over cap and (b) over cap.