Graphene's high mobility and Fermi velocity, combined with its constant light absorption in the visible to far-infrared range, make it an ideal material to fabricate high-speed and ultrabroadband photodetectors. However, the precise mechanism of photodetection is still debated. Here, we report wavelength and polarization-dependent measurements of metal-graphene-metal photodetectors. This allows us to quantify and control the relative contributions of both photothermo- and photoelectric effects, both adding to the overall photoresponse. This paves the way for a more efficient photodetector design for ultrafast operating speeds.