Global changes in land use are threatening the diversity of many ecosystems on both the intra- and interspecific levels. Among these ecosystems are the species-rich hay meadows, which have drastically declined in quality and quantity, due to land use intensification or abandonment in recent decades. The remaining genetic resources of their plant species must therefore be protected. To determine the driving forces impacting genetic variation in common hay meadow species (Dactylis glomerata, Heracleum sphondylium, and Trifolium pratense), we used data on the land use history, historic and present landscape structure and habitat quality. Our results showed average genetic diversity within the study sites, with low differentiation levels and a high gene flow among grasslands. Land use history, landscape structure and habitat quality were found to be related to the distribution of genetic diversity in the studied species, highlighting the complex forces acting in these ecosystems and showing the specific impact of litter accumulation on genetic diversity. Both historic and current environmental variables influence genetic diversity, demonstrating the importance of the land use history of a habitat. The most important group of variables impacting genetic variation in all three species was the landscape structure (e.g., distance to the nearest-located urban area or grassland). Also important was the influence of litter cover on genetic diversity in D. glomerata, which provides an interesting starting point for further research.