The two-phonon infrared absorption spectrum of the covalent elemental semiconductors Si and Ge is calculated completely ab initio. Besides the harmonic phonon eigen solutions this involves the phonon-photon coupling constants, the so called second-order dipole moments (also called dipole coefficients). These are given by the third-order derivatives of the total energy with respect to an electric field (once) and to atomic displacements (twice). In the framework of density functional theory, we have applied the 2n+1 theorem to derive an analytic expression for the second-order dipole moments. Numerical calculations of these coefficients and of the infrared absorption spectrum of Si and Ge are carried out. The shape and overall intensities of the spectra compare well with experimental data, even though some discrepancies remain.