We consider phase-coherent transport through ballistic and diffusive two-dimensional hole systems based on the Kohn-Luttinger Hamiltonian. We show that intrinsic heavy-hole-light-hole coupling gives rise to clear-cut signatures of an associated Berry phase in the weak localization which renders the magnetoconductance profile distinctly different from electron transport. Nonuniversal classical correlations determine the strength of these Berry phase effects and the effective symmetry class, leading even to antilocalization-type features for circular quantum dots and Aharonov-Bohm rings in the absence of additional spin-orbit interaction. Our semiclassical predictions are confirmed by numerical calculations.