Background: Health promoters often use stakeholder groups to jointly plan and implement local interventions. Stakeholder groups should take over responsibility to later run the health promotion program independently. Monitoring this process of capacity building can help health promoters improve the quality of the process. Instruments for the systematic assessment of capacity building among stakeholder groups are scarce. The goal of this study was to develop, and pilot test a generic assessment instrument for setting-based capacity building. Methods: We drafted a semi-standardized monitoring instrument to be used in stakeholder groups in various settings. This "EVA-protocol" (short for evaluation protocol) was based on capacity building domains e.g., leadership, resource mobilization. It was pilot implemented in a research network on increasing an active lifestyle in various settings. The respective health promoters documented 78 meetings of 15 different stakeholder groups. We performed feedback interviews and member checking among the facilitating health promoters, asking for comprehensibility, length, usability and perceived benefits of the instrument. Findings: Data collected in the "EVA-protocol" helped the facilitating health promoters understand the development of competences and capacities in the stakeholder groups and identify factors that favor or hinder the capacity building process. The instrument was rated as user friendly, but it was remarked that it is best filled out by two persons and reflected upon by those to offer the greatest benefit. Not all projects could afford this procedure due to lack of time/staff resources. Conclusions: The drafted instrument can serve as quality management tool for health promoters who facilitate participatory stakeholder groups in different settings and intend to build capacities for sustainable health promotion structures.