Light-matter interactions are governed by conservation laws of energy and momentum. For harmonic generation in crystalline solids, energy conservation imposes that m incoming photons with energy ℏω0 are combined to form one photon at energy mℏω0. Linear momentum conservation governs phase matching, whereas angular momentum conservation connects the angular momentum carried by photons to the discrete rotational symmetry of the crystal lattice. As a consequence, circular harmonic generation exerts a torque on the lattice and, conversely, a macroscopic rotation of the crystal induces a nonlinear rotational Doppler shift. These cornerstone laws of nonlinear optics rely on macroscopic symmetry arguments, and therefore provide little insight into the microscopic origin of angular momentum transfer. Here we uncover a direct connection between angular momentum conservation in nonlinear optics and the electronic quantum geometry, by proving that the transferred angular momentum from light to the crystal is proportional to the local Berry curvature at one optical resonance. This relation is encoded in the nonlinear harmonic circular dichroism, which we measure experimentally in an atomically thin semiconductor. With this, we extend our understanding of nonlinear optics, and we establish a method for the all-optical control and read-out of the local Berry curvature.