The anharmonic decay rates of atomic vibrations in amorphous silicon (a-Si) and paracrystalline silicon (p-Si), containing small crystalline grains embedded in a disordered matrix, are calculated using realistic structural models. The models are 1000-atom four-coordinated networks relaxed to a local minimum of the Stillinger-Weber interatomic potential. The vibrational decay rates are calculated numerically by perturbation theory, taking into account cubic anharmonicity as the perturbation. The vibrational lifetimes for a-Si are found to be on picosecond time scales, in agreement with the previous perturbative and classical molecular dynamics calculations on a 216-atom model. The calculated decay rates for p-Si are similar to those of a-Si. No modes in p-Si reside entirely on the crystalline cluster, decoupled from the amorphous matrix. The localized modes with the largest (up to 59%) weight on the cluster decay primarily to two diffusons. The numerical results are discussed in relation to a recent suggestion by van der Voort et al. [Phys. Rev. B 62, 8072 (2000)] that long vibrational relaxation inferred experimentally may be due to possible crystalline nanostructures in some types of a-Si.