Using the Kubo linear response formalism, we study the effects of intrinsic graphene optical and surface polar phonons (SPPs) on the optical conductivity of doped graphene. We find that inelastic electron-phonon scattering contributes significantly to the phonon-assisted absorption in the optical gap. At room temperature, this midgap absorption can be as large as 20-25% of the universal ac conductivity for graphene on polar substrates (such as Al2O3 or HfO2) due to strong electron-SPP coupling. The midgap absorption, moreover, strongly depends on the substrates and doping levels used. With increasing temperature, the midgap absorption increases, while the Drude peak, on the other hand, becomes broader as inelastic electron-phonon scattering becomes more probable. Consequently, the Drude weight decreases with increasing temperature. DOI: 10.1103/PhysRevB.87.035414