Cyber security exercises are an essential means to train people and increase their skill levels in IT operations, cyber incident response, and forensic investigations. Unfortunately, carrying out high-quality exercises requires tremendous human effort in planning, deploying, executing and evaluating well-planned cyber exercise scenarios. While planning a scenario is often only a one time effort, and deployment can be highly automatized today, their repeated execution and evaluation is a resource-intensive task. Usually human experts manually observe the participants to recognize any difficulties in carrying out the exercise and to keep track of the participants’ progress. This is an essential prerequisite to not only support participants during the exercise, but also to drive the scenario further through timely injects, and provide feedback after the exercise. All this manual effort makes exercises a costly activity, reduces scalability and hinders their wide adoption. We argue that with automating observations, recognizing participant progress with only little to no human effort, and even steering the delivery of customized injects, cyber exercises could be carried out much more cost-effective. In this paper, we therefore introduce the concept of monitoring points which enable the scenario-dependent collection of technical data and the calculation of behavior and progress metrics to rate participants in exercises. This is the foundational basis for steering an exercise on the one side, and evaluation on the other side. We showcase our concept and implementation in course of a demonstrator consisting of a cyber exercise comprising 14 participants and discuss its applicability.