Users’ expectations about systems alter how they interact with them, thereby influencing their experience. Latency is also known to alter experience and performance in interactive systems, particularly in video games. Currently, it is unclear if users’ expectations influence latency-based effects. We report the results of an experiment (N = 24) in which participants played a video game with four levels of phantom latency (30 ms, 60 ms, 90 ms, and 120 ms). Crucially, all rounds were played with 75 ms of actual latency, we merely changed its presentation in the game. We show that a high phantom latency reduces game experience and performance. Participants were least accurate and effective when playing with 120 ms while feeling the least competent and tensest. We concluded that the effects of latency are partly expectation-based. Latency researchers, developers, and gamers must be aware of the effects induced by the expectation of latency.