Despite considerable progress during the last decades in devising a semiclassical theory for classically chaotic quantum systems a quantitative semiclassical understanding of their dynamics at late times (beyond the so-called Heisenberg time T_H) is still missing. This challenge, corresponding to resolving spectral structures on energy scales below the mean level spacing, is intimately related to the quest for semiclassically restoring quantum unitarity, which is reflected in real-valued spectral determinants. Guided through insights for quantum graphs we devise a periodic-orbit resummation procedure for quantum chaotic systems invoking periodic-orbit self encounters as the structuring element of a hierarchical phase space dynamics. We propose a way to purely semiclassically construct real spectral determinants based on two major underlying mechanisms: (i) Complementary contributions to the spectral determinant from regrouped pseudo orbits of duration T<T_H and T_H−T are complex conjugate to each other. (ii) Contributions from long periodic orbits involving multiple traversals along shorter orbits cancel out. We furthermore discuss implications for interacting N-particle quantum systems with a chaotic classical large-N limit that have recently attracted interest in the context of many-body quantum chaos.