Electron interferometry via phase-contrast microscopy, holography, or picodiffraction can provide a direct visualization of the static electric and magnetic fields inside or around a material at subatomic precision, but understanding the electromagnetic origin of light-matter interaction requires time resolution as well. Here, we demonstrate that pump-probe electron diffraction with all-optically compressed electron pulses can capture dynamic electromagnetic potentials in a nanophotonic material with sub-light-cycle time resolution via centrosymmetry-violating Bragg spot dynamics. The origin of this effect is a sizable quantum mechanical phase shift that the electron de Broglie wave obtains from the oscillating electromagnetic potentials within less than 1 fs. Coherent electron imaging and scattering can therefore reveal the electromagnetic foundations of light-matter interaction on the level of the cycles of light.