Electronic transport through chaotic quantum dots exhibits universal, system independent, properties, consistent with random matrix theory. The quantum transport can also be rooted, via the semiclassical approximation, in sums over the classical scattering trajectories. Correlations between such trajectories can be organized diagrammatically and have been shown to yield universal answers for some observables. Here, we develop the general combinatorial treatment of the semiclassical diagrams, through a connection to factorizations of permutations. We show agreement between the semiclassical and random matrix approaches to the moments of the transmission eigenvalues. The result is valid for all moments to all orders of the expansion in inverse channel number for all three main symmetry classes (with and without time reversal symmetry and spin-orbit interaction) and extends to nonlinear statistics. This finally explains the applicability of random matrix theory to chaotic quantum transport in terms of the underlying dynamics as well as providing semiclassical access to the probability density of the transmission eigenvalues.