The Wigner time delay, defined by the energy derivative of the total scattering phase shift, is an important spectral measure of an open quantum system characterizing the duration of the scattering event. It is proportional to the trace of the Wigner-Smith matrix Q that also encodes other time-delay characteristics. For chaotic cavities, these quantities exhibit universal fluctuations that are commonly described within random matrix theory. Here, we develop a new semi-classical approach to the time-delay matrix which is formulated in terms of the classical trajectories that connect the exterior and interior regions of the system. This approach is superior to previous treatments because it avoids the energy derivative. We demonstrate the method's efficiency by going beyond previous work in establishing the universality of time-delay statistics for chaotic cavities with perfectly connected leads. In particular, the moment generating function of the proper time-delays (eigenvalues of Q) is found semiclassically for the first five orders in the inverse number of scattering channels for systems with and without time-reversal symmetry. We also show the equivalence of random matrix and semiclassical results for the second moments and for the variance of the Wigner time delay at any channel number.