Phonon-induced orbital and spin relaxation rates of single electron states in lateral single and double quantum dots are obtained numerically for realistic materials parameters. The rates are calculated as a function of magnetic field and interdot coupling, at various field and quantum dot orientations. It is found that orbital relaxation is due to deformation potential phonons at low magnetic fields, while piezoelectric phonons dominate the relaxation at high fields. Spin relaxation, which is dominated by piezoelectric phonons, in single quantum dots is highly anisotropic due to the interplay of the Bychkov-Rashba and Dresselhaus spin-orbit couplings. Orbital relaxation in double dots varies strongly with the interdot coupling due to the cyclotron effects on the tunneling energy. Spin relaxation in double dots has an additional anisotropy due to anisotropic spin hot spots which otherwise cause giant enhancement of the rate at useful magnetic fields and interdot couplings. Conditions for the absence of the spin hot spots in in-plane magnetic fields (easy passages) and perpendicular magnetic fields (weak passages) are formulated analytically for different growth directions of the underlying heterostructure. It is shown that easy passages disappear (spin hot spots reappear) if the double dot system loses symmetry by an xy-like perturbation.