CO-terminated tips currently provide the best spatial resolution obtainable in atomic force microscopy. Due to their chemical inertness, they allow us to probe interactions dominated by Pauli repulsion. The small size and inertness of the oxygen front atom yields unprecedented resolution of organic molecules, metal clusters, and surfaces. We study the capability of CO-terminated tips to laterally manipulate single iron adatoms on the Cu(111) surface with combined atomic force and scanning tunneling microscopy at 7 K. We find that even a slight asymmetry of the tip results in a distortion of the lateral force field. In addition, the influence of the tilt of the CO tip on the lateral force field is reversed compared to the use of a monoatomic metal tip which we can attribute to the inverted dipole moment of a CO tip with respect to a metal tip. Moreover, we demonstrate atom-by-atom assembly of iron clusters with CO tips while using the high-resolution capability of the CO tips in between to determine the arrangement of the individual iron atoms within the cluster. In all conducted experiments using CO tips within this study, the CO was never changed or lost from the tip's apex.