Many studies and experiments suggest the muscarinic M5 receptor (M5R) as a potential target for the clinical treatment of drug abuse, schizophrenia, or Alzheimer’s disease. In the present study, a library containing 39 novel hM5R antagonists based on an arecaidine (tetrahydropyridine) core was synthesized, pharmacologically characterized in radioligand competition binding or mini-G protein assays and analyzed in molecular docking studies to elucidate their binding modes and in vitro characteristics. Compound 56m showed low single-digit nanomolar affinity on hM5R combined with a 17-fold selectivity over hM2R, while compound 81d displayed good affinity on hM5R with a 3- to 12-fold selectivity over the other subtypes. Induced fit docking studies suggest the conserved Y2.60 and hM5R-unique Q45.51 as the responsible amino acids regarding affinity and selectivity.