Mainz hit back as they won a free kick on the edge of the box and Christian Clemens lined up a strike, which he lifted over the wall but also just over the crossbar.
In the second half, Mainz pushed forward again and Yunus Malli found his way through on goal, but Roos made a good save with his body to keep the scores level.
The German team took the lead in the 72nd minute when the Millers failed to clear their lines from a cross and Leon Balogun collected the ball and fired it into the net.
Rotherham almost levelled immediately as Aidy White found space to curl an effort towards goals, which took a deflection, but the ball cannoned off the post.
The home side did get their equaliser when Derbyshire received the ball down the left wing and then cut inside onto his right foot to curl an effort beyond the keeper.
The Millers nicked the win in the 89th minute when Bowery latched on to the ball in the box and buried his strike into the bottom corner.