We study the implications of unitarity for pseudo-orbit expansions of the spectral determinants of quantum maps and quantum graphs. In particular, we advocate to group pseudo-orbits into subdeterminants. We show explicitly that the cancellation of long orbits is elegantly described on this level and that unitarity can be built in using a simple subdeterminant identity which has a nontrivial interpretation in terms of pseudo-orbits. This identity yields much more detailed relations between pseudo-orbits of different lengths than was known previously. We reformulate Newton identities and the spectral density in terms of subdeterminant expansions and point out the implications of the subdeterminant identity for these expressions. We analyze furthermore the effect of the identity on spectral correlation functions such as the autocorrelation and parametric cross-correlation functions of the spectral determinant and the spectral form factor.