The Drude-Smith model has been used extensively in fitting the THz conductivities of nanomaterials with carrier confinement on the mesoscopic scale. Here, we show that the conventional "backscattering" explanation for the suppression of low-frequency conductivities in the Drude-Smith model is not consistent with a confined Drude gas of classical noninteracting electrons and we derive a modified Drude-Smith conductivity formula based on a diffusive restoring current. We perform Monte Carlo simulations of a model system and show that the modifiedDrude-Smith model reproduces the extracted conductivitieswithout free parameters. This alternate route to the Drude-Smith model provides the popular formula with a more solid physical foundation and well-defined fit parameters.