(1) Background: Keratinocyte cancer (KC) is associated with exposure to ultraviolet (UV) radiation. However, data are controversial as to whether chronic UV exposure or high intermittent UV exposure are key drivers of carcinogenesis in cutaneous squamous cell carcinoma (cSCC) and basal cell carcinoma (BCC). Prolonged sun exposure of the skin causes photo-aging, which is associated with actinic elastosis, a condition characterized by the degeneration of elastin in the upper dermis, which is assessable via conventional histology. In this study, we aimed to compare the degree of actinic elastosis in different types of KC with regard to various patient characteristics. (2) Methods: We defined a semiquantitative score for the degree of actinic elastosis ranging from 0 = none to 3 = total loss of elastic fibers (basophilic degeneration). The extent was measured histometrically by two independent dermatohistopathologists in the immediate vicinity of 353 KC. The scores were merged and matched with tumor types (cSCC and BCC with subtypes), and clinical variables such as body site, sex and age. (3) Results: As expected, the degree of actinic elastosis correlated with age. However, it was significantly higher in cSCC compared to BCC irrespective of age, sex, body site and tumor subtypes. (4): Conclusions: Lifetime sun exposure may be estimated via routine histology using this scoring technique for actinic elastosis as a surrogate marker. cSCCs are more strongly associated with chronic UV exposure than BCCs, even in sun-exposed localizations such as the face.