The League Two relegation strugglers send the Conference side out of the competition, courtesy of a second-half Max Power penalty.
After a goalless first half, Tranmere broke the deadlock 10 minutes into the second half, as Mark McChrystal hand-balled inside his own area, which Power converted into the top left-hand corner.
The League Two side started to cruise but suffered a setback as George Donnelly was shown a straight red card for fighting.
Tranmere remained in control but the red card gave the non-league side a glimmer of hope.
The League Two side saw out the remainder of the game to end Bristol Rovers' cup run.