Geometric-phase backaction in a mesoscopic qubit-oscillator system