We consider a qubit coupled to a nonlinear quantum oscillator, the latter coupled to an Ohmic bath, and investigate the qubit dynamics. This composed system can be mapped onto that of a qubit coupled to an effective bath. An approximate mapping procedure to determine the spectral density of the effective bath is given. Specifically, within a linear response approximation the effective spectral density is given by knowledge of the linear susceptibility of the nonlinear quantum oscillator. To determine the actual form of the susceptibility, we consider its periodically driven counterpart, which is the problem of the quantum Duffing oscillator within linear response theory in the driving amplitude. Knowing the effective spectral density, the qubit dynamics is investigated. In particular, an analytic formula for the qubit's population difference is derived. Within the regime of validity of our theory, a very good agreement is found with predictions obtained from a Bloch-Redfield master equation approach applied to the composite qubit-nonlinear oscillator system.