Quantum geometry and topology are fundamental concepts of modern condensed matter physics, underpinning phenomena ranging from the quantum Hall effect to protected surface states. The Berry curvature, a central element of this framework, is well established for its key role in electronic transport, whereas its impact on the optical properties of crystals remains comparatively unexplored. Here, we derive a relation between optical activity, defined by the gyration tensor, and the k-derivatives of the Berry curvature at optical resonances in the Brillouin zone. We systematically determine which of these derivatives are non-zero or constrained by symmetry across all time-reversal-invariant crystal classes. In particular, we analytically demonstrate that circular dichroism emerges in chiral crystal classes as a result of a non-zero Berry curvature k-derivative along the optical axis, and we interpret this finding based on the conservation of angular momentum in light-matter interactions. This work establishes a quantum-geometric framework for optical activity in solids and it opens new routes to probe quantum geometry via linear and nonlinear optics.