Oxford took the tie the distance with two goals in a frenetic final six minutes of normal time but Saint-Maximin's strike was enough to settle the contest 3-2 in favour of Steve Bruce's side.
Recent cup runs have been lacking for Newcastle but they would have thought they were on a smooth course to round five as Sean Longstaff and Joelinton had them two goals up at half-time of a replay which stemmed from a forgettable 0-0 draw at St James' Park.
Karl Robinson's side were the livelier of the two as the additional 30 minutes got under way but, after a few miscues from his team-mates, Saint-Maximin provided the heroics.
Newcastle, showing five changes from their weekend Premier League stalemate at home to Norwich, struck with their first clear chance on goal.
Joelinton's hold-up play was good and he rolled a pass to Sean Longstaff, who controlled the ball on the edge of the box and finished well to give Oxford goalkeeper Simon Eastwood no chance.
But, while Joelinton is adept at linking up with his team-mates, his own finishing remains questionable and he missed a sitter just a minute later – although he soon made sure he was on the scoresheet.
Oxford finally responded and were unlucky not to peg their visitors back immediately, first Jamie Mackie forcing a fine stop from Karl Darlow before Sam Long's follow-up was headed off the line by Jamaal Lascelles.
Joelinton had gone down injured just before his goal and was replaced by Saint-Maximin, with Almiron missing a decent chance to further widen the gap soon after.
There was little goalmouth action in an even second half until Kelly stood up to strike a sweet free-kick, his first goal for Oxford, with six minutes remaining and Newcastle hearts would have been in mouths as full-back Josh Ruffels blazed over in the dying stages.
But the tie would require another 30 minutes to be settled after Holland broke his Oxford duck with a fine effort deep into added-time.
The hosts maintained their momentum in extra-time but there was to be no dream cup upset as Saint-Maximin struck to ensure Newcastle will travel to West Brom next month.