We map out the band structure of a one-dimensional photonic crystal while a 12-fs control pulse activates ultrastrong interaction with quantized electronic transitions in semiconductor quantum wells. Phase-locked multi-terahertz transients trace the buildup of a large vacuum Rabi splitting and an unexpected asymmetric formation of the upper and lower polariton bands. The pronounced flattening of the photonic bands causes a slow-down of the group velocity by one order of magnitude on the time scale of the oscillation period of light.