The increasing availability and deployment of open source software in personal and commercial environments makes open source software highly appealing for hackers, and others who are interested in exploiting software vulnerabilities. This deployment has resulted in a debate “full of religion” on the security of open source software compared to that of closed source software. However, beyond such arguments, only little quantitative analysis on this research issue has taken place. We discuss the state-of-the-art of the security debate and identify shortcomings. Based on these, we propose new metrics, which allows to answer the question to what extent the review process of open source and closed source development has helped to fix vulnerabilities. We illustrate the application of some of these metrics in a case study on OpenOffice (open source software) vs. Microsoft Office (closed source software).