Topology is fundamental to condensed matter physics, underlying phenomena such as the quantum Hall effect in its various forms, protected surface states and spin-valley locking. Central to this is the Berry curvature, a measure of the geometric properties of Bloch wavefunctions in crystals. Despite its established connections to space inversion and time-reversal symmetries, a comprehensive framework linking the Berry curvature to other crystal symmetries is still missing. Here, we present a theoretical model for the Jacobian matrix of the Berry curvature in the proximity of optical resonances for all non-magnetic crystal classes. The Jacobian matrix of the Berry curvature is readily derived from the second-order nonlinear optical susceptibilities of the time-invariant point groups. We show that the diagonal elements of the Jacobian matrix, which define the divergence of the Berry curvature, are non-zero only for the chiral crystal classes, pointing towards a topological origin of circular dichroism. In polar crystals, we uniquely find a non-zero curl of the Berry curvature, suggesting a topological origin of spontaneous polarization via the Maxwell-Berry equations. Our findings thus open new perspectives for the topological interpretation of textbook phenomena at the heart of condensed matter physics.