Latency is inherently part of every interactive computing system and particularly important for video games. Previous work shows that constant latency above 25 ms reduces game experience and player performance. However, latency in the wild varies and is never constant due to multiple factors, such as updates in routing tables, users changing their location, or the system's workload. It is unclear if switching latency impairs the gaming experience stronger than a constant high latency. To elucidate, we conducted an experiment with 264 participants playing a shooting video game induced with 0 ms, 33 ms, and 66 ms controlled latency. While playing, the game switched between different latency levels based on three frequencies. Our analysis shows that switching latency significantly impaired the participants' flow. Additionally, we found effects on the perceived tension, the experienced challenge, and the players' performance. We conclude that games should prioritize constant latency, even if that entails artificially adding latency.