In interactive systems high latency affects user performance and experience. This is especially problematic in video games. A large number of studies on this topic investigated the effects of constant, high latency. However, in practice, latency is never constant but varies by up to 100 ms due to variations in processing time and delays added by polling between system components. In a large majority of studies, these variations in latency are neither controlled for nor reported. Thus, it is unclear to which degree small, continuous variations in latency affect user performance. If these unreported variations had a significant impact, this might cast into doubt the findings of some studies. To investigate how latency variation affects player performance and experience in games, we conducted an experiment with 28 participants playing a first-person shooter. Participants played with two levels of base latency (50 ms vs. 150 ms) and variation (0 ms vs. 50 ms). As expected, high base latency significantly reduces player performance and experience. However, we found strong evidence that small variations in latency in the order of 50 ms, do not affect player performance significantly. Thus, our findings mitigate concerns that previous latency studies might have systematically ignored a confounding effect.