Despite considerable progress during the past decades in devising a semiclassical theory for classically chaotic quantum systems a quantitative semiclassical understanding of their dynamics at late times (beyond the so-called Heisenberg time T_H) is still missing. This challenge, corresponding to resolving spectral structures on energy scales below the mean level spacing, is intimately related to the quest for semiclassically restoring unitary quantum evolution. Guided through insights for quantum graphs we devise a periodic-orbit resummation procedure for spectra of quantum chaotic systems invoking periodic-orbit self-encounters as the structuring element of a hierarchical phase space dynamics. Quantum unitarity is reflected in real-valued spectral determinants with zeros giving discrete energy levels. We propose a way to purely semiclassically construct such real spectral determinants based on two major underlying mechanisms. (i) Complementary contributions to the spectral determinant from regrouped pseudo-orbits of duration T < T_H and T_H − T are complex conjugate to each other. (ii) Contributions from long periodic orbits involving multiple traversals along shorter orbits cancel out. We furthermore discuss implications for interacting N-particle quantum systems with a chaotic classical large-N limit that have recently attracted particular interest in the context of many-body quantum chaos.