We investigate theoretically the magnetic-field-dependent far-infrared response of holes confined in a quantum dot. In contrast to quantum dots with electrons, the mixing of heavy- and light-hole states prohibits a separation of the center-of-mass motion from the relative motion. This violation of the condition required to satisfy the generalized Kohn's theorem leads to a calculated absorption spectrum that is considerably more complex than the corresponding spectrum for quantum dots containing electrons.