Quantum cavities or dots have markedly different properties depending on whether their classical counterparts are chaotic or not. Connecting a superconductor to such a cavity leads to notable proximity effects, particularly the appearance, predicted by random matrix theory, of a hard gap in the excitation spectrum of quantum chaotic systems. Andreev billiards are interesting examples of such structures built with superconductors connected to a ballistic normal metal billiard since each time an electron hits the superconducting part it is retroreflected as a hole (and vice versa). Using a semiclassical framework for systems with chaotic dynamics, we show how this reflection, along with the interference due to subtle correlations between the classical paths of electrons and holes inside the system, is ultimately responsible for the gap formation. The treatment can be extended to include the effects of a symmetry-breaking magnetic field in the normal part of the billiard or an Andreev billiard connected to two phase-shifted superconductors. Therefore, we are able to see how these effects can remold and eventually suppress the gap. Furthermore, the semiclassical framework is able to cover the effect of a finite Ehrenfest time, which also causes the gap to shrink. However, for intermediate values this leads to the appearance of a second hard gap-a clear signature of the Ehrenfest time.