The complex structure of the valence band in many semiconductors leads to multifaceted and unusual properties for spin-3/2 hole systems compared to common spin-1/2 electron systems. In particular, two-dimensional hole systems show a highly anisotropic Zeeman interaction. We have investigated this anisotropy in GaAs/AlAs quantum well structures both experimentally and theoretically. By performing time-resolved Kerr rotation measurements, we found a nondiagonal tensor g that manifests itself in unusual precessional motion, as well as distinct dependencies of hole-spin dynamics on the direction of the magnetic field B. We quantify the individual components of the tensor g for [113]-, [111]-, and [110]grown samples. We complement the experiments by a comprehensive theoretical study of Zeeman coupling in in-plane and out-of-plane fields B. To this end, we develop a detailed multiband theory for the tensor g. Using perturbation theory, we derive transparent analytical expressions for the components of the tensor g that we complement with accurate numerical calculations based on our theoretical framework. We obtain very good agreement between experiment and theory. Our study demonstrates that the tensor g is neither symmetric nor antisymmetric. Opposite off-diagonal components can differ in size by up to an order of magnitude. The tensor g encodes not only the Zeeman energy splitting but also the direction of the axis about which the spins precess in the external field B. In general, this axis is not aligned with B. Hence our study extends the general concept of optical orientation to the regime of nontrivial Zeeman coupling.