Interactive tabletops do not only offer a large surface for collaborative interaction. They also offer quick access to digital tools directly at the table - where a large number of everyday activities take place. Tabletops with an embedded display are generally less flexible and more fragile than ordinary massive tabletops. Physical objects on the tabletop occlude the digital content. In contrast, top-down-projected interfaces using an overhead camera-projector system allow for augmenting arbitrary tables and the object lying on them. However, detecting pointing input only via a camera image captured from above requires robustly recognizing whether a finger or pen touches the tabletop or whether it hovers slightly above it. In this paper, we present a solution for reliably tracking a pen on arbitrary tabletop surfaces. The pen emits infrared light via a tip made of optical fiber. A camera captures position and shape of the light point on the surface. Our open-source tracking algorithm combines heuristics and a machine learning model to distinguish between drawing and hovering. A pilot study with 7 participants shows that that this system can be reliably used for drawing and writing on tabletops. However, occlusion by users’ hands can deteriorate tracking of the pen.