Now they are aiming for a fifth – and which would be the biggest yet – with Manchester City due to visit Rodney Parade on February 16.
Here, Press Association Sport looks back at their proud recent record under manager Michael Flynn.
Newport 2 Leeds 1 – January 7 2018
Shawn McCoulsky, on loan from Bristol City, wrote his name into Newport's history books with an 89th-minute winner against the Sky Bet Championship visitors in this third-round tie. It completed a late turnaround for Newport, who had trailed to Gaetano Berardi's ninth-minute strike. But Newport were level through a 76th-minute own goal from Conor Shaughnessy, inadvertently turning home Frank Nouble's cross, and Leeds' misery was complete with a late red card for Samuel Saiz.
Newport 1 Tottenham 1 – January 27 2018
Newport's reward for beating Leeds was a fourth-round home tie with Premier League big boys Tottenham. And they were in dreamland again seven minutes before the break when Padraig Amond headed them in front. Hopes grew as Newport kept Tottenham at bay for much of the second half, but Harry Kane broke County hearts eight minutes from time. The draw earned Newport a lucrative replay at Wembley, though their FA Cup adventure went no further as they lost 2-0 under the arch.
Newport 2 Leicester 1 – January 6, 2019
Newport's biggest scalp yet was claimed in the third round of this season's competition when 2016 Premier League champions Leicester – 74 places above Newport in the league at the time – were dumped out. Newport took an early lead with a well-worked goal, finished by Jamille Matt, and Leicester were frustrated by the crossbar and a string of saves from Joe Day. Rachid Ghezzal appeared to have rescued a replay for the Foxes in the 82nd minute, but Marc Albrighton's handball gifted Newport a penalty three minutes later and Amond made no mistake from the spot to secure a famous win.
Newport 2 Middlesbrough 0 – February 5, 2019
After securing a dramatic 1-1 draw at the Riverside Stadium thanks to Matthew Dolan's stoppage-time goal, Newport welcomed Tony Pulis' side to their manager's home town for a replay. With the prestige of a fifth-round tie with Manchester City up for grabs, Newport were good value for their win, which was secured via second-half goals from Robbie Willmott and that man Amond again. A joyous night for the club was completed after the final whistle, with goalkeeper Day sprinting off the pitch to join his wife following the birth of their twins.