The vortex state, characterized by a curling magnetization, is one of the equilibrium configurations of soft magnetic materials(1-4) and occurs in thin ferromagnetic square and disk-shaped elements of micrometre size and below. The interplay between the magneto-static and the exchange energy favours an in-plane, closed flux domain structure. This curling magnetization turns out of the plane at the centre of the vortex structure, in an area with a radius of about 10 nanometres-the vortex core(5-7). The vortex state has a specific excitation mode: the in-plane gyration of the vortex structure about its equilibrium position(8-10). The sense of gyration is determined by the vortex core polarization(11). Here we report on the controlled manipulation of the vortex core polarization by excitation with small bursts of an alternating magnetic field. The vortex motion was imaged by time-resolved scanning transmission X-ray microscopy(12). We demonstrate that the sense of gyration of the vortex structure can be reversed by applying short bursts of the sinusoidal excitation field with amplitude of about 1.5 mT. This reversal unambiguously indicates a switching of the out-of-plane core polarization. The observed switching mechanism, which can be understood in the framework of micromagnetic theory, gives insights into basic magnetization dynamics and their possible application in data storage.