Based on general symmetry considerations, we investigate how the dependence of the tunneling anisotropic magnetoresistance (TAMR) on the magnetization direction is determined by the specific form of the spin-orbit coupling field. By extending a phenomenological model, previously proposed for explaining the main trends of the TAMR in (001) ferromagnet/semiconductor/normal-metal magnetic tunnel junctions (MTJs) [J. Moser et al., Phys. Rev. Lett. 99, 056601 (2007)], we provide a unified qualitative description of the TAMR in MTJs with different growth directions. In particular, we predict the forms of angular dependences of the TAMR in (001), (110), and (111) MTJs with structure inversion asymmetries and/or bulk inversion asymmetries. The effects of in-plane uniaxial strain on the TAMR are also investigated.