Access reviews, i.e. the periodical security audit of access privileges, are a basic compliance and IT-security requirement for medium- and large-scale organizations. Assessing the quality of the reviewer's decisions ex-post can help to analyse the effectiveness of the measure and to identify structural or organizational shortcomings. Yet, current studies merely focus on improving the decision-making process itself. This paper develops a method for assessing the decision quality of access reviews by applying a solution from the crowd sourcing research realm. In order to achieve this, the problem of assessing decision quality of access reviews is generalized. It is shown that the abstract problem can be mapped to the problem of assessing the quality of crowd tagging decisions. Subsequently, an applicable solution of this research area is applied to access reviews. Furthermore, the selected approach is optimized to meet the specific challenges of access review data.