Mikel Arteta made seven changes to his side and was without captain Pierre-Emerick Aubameyang for personal reasons as the Gunners were outplayed, losing their grip on the trophy following a 1-0 defeat at St Mary's.
Southampton should have won by a bigger margin but now face a trip to Wolves in round five as Gabriel's poor return to the Arsenal side was capped by accidentally deciding the tie in favour of the hosts.
Saints started lively and were unsettling the recalled Gabriel in the opening minute.
But the visitors still looked frail at the back as Che Adams burst past Gabriel as if he was not there before forcing a smart stop out of Leno.
Gabriel Martinelli then passed up a fine opportunity to put the 14-time FA Cup winners in front, miscuing his shot after breaking the offside trap from a free-kick.
Saints took the lead just before the midway point of the first half as Elneny gave away possession on the edge of his own box.
The ball was worked wide to Kyle Walker-Peters, whose cross-cum-shot was deflected into his own goal by the unfortunate Gabriel, the first time Arsenal have conceded in 508 minutes.
The second half started in similar fashion, Arsenal shabby in possession as Southampton pressed the wobbly backline.
But it was defender Rob Holding who spurned a great chance to equalise, sending an effort over the crossbar after the ball bobbled up for him in the Southampton box.
The shift meant Arsenal enjoyed more possession in and around the Southampton box without ever really looking like finding the goal to take the tie into extra-time.