The Championship side were being outplayed by their League One opponents when Martin produced a magnificent right-footed shot on the turn from the edge of the box on 83 minutes for what proved to be the winner.
Even then Portsmouth refused to lie down and were denied an equaliser with the last kick of the game when Bentley spread himself to keep out a close-range shot from substitute Jordy Hiwula-Mayifuila.
Famara Diedhiou had fired City into a 19th-minute lead with a left-footed strike from just outside the box after latching onto Hakeeb Adelakun's pass.
But Pompey levelled in first half stoppage time, Andrew Cannon pulling the ball back from the left and right-back Callum Johnson drilling a low drive past Bentley.
Holden said: "I was pleased with our first-half display, but we couldn't maintain it after the break and needed some moments of quality.
"Chris Martin's strike was a moment of pure genius and Dan Bentley's save at the end showed his importance to us. I wouldn't swap him for any keeper in the EFL or above.
"I fielded a strong side because I want to progress in the competition and knew Portsmouth would present a tough test.
"It was great to have Alfie Mawson back in defence after his injury lay-off and the intention was always to play him for an hour.
"In the first half we passed the ball really well and scored an excellent goal, only for them to equalise with their first shot on target.
"After the interval we gave away the ball too often and allowed them to hit us on the break. There is work to do before our next game at Brentford."
Pompey boss Kenny Jackett was in no mood for hard luck stories.
"We have caused a very good Championship side a lot of problems today, but that is no good if you don't turn pressure into goals," he said.
"The difference was in the quality of the finishing. I was right behind Martin's shot and it was a terrific strike.
"We are disappointed to lose a game in which our approach play was so good. A mixture of brave defending, good saves and failure to take our chances has cost us.
"Bentley's save at the end was a really good one because Jordy struck his shot well."
Ronan Curtis was missing from the Portsmouth team because of illness and Jackett made it clear he would be tested for Covid-19.
"Ronan had a negative test during the week, but he felt under the weather, so we didn't bring him with us," he said.
"We will probably get him tested again tomorrow, but hopefully it is nothing serious."