The Saints were rewarded for their attacking intent in the opening period when Gabriel turned in Kyle Walker-Peters' cross in the 24th minute.
It was the first goal Arsenal had conceded in 508 minutes as they suffered a first loss in seven matches.
Southampton will face another Premier League side, Wolves, in the next round.
With their next opponents enduring a poor run in the league and having struggled in their 1-0 win at non-league Chorley on Friday, the 1976 winners will fancy their chances of progressing further at Molineux.
The Saints also went close through Che Adams, whose low, angled strike was saved brilliantly by Bernd Leno, and James Ward-Prowse, who saw his delivery from a corner come off the bar.
Mikel Arteta's Gunners side were tepid as an attacking force during the first 45 minutes, but looked more of a threat after the break.
However, Eddie Nketiah's deflected shot in the 67th minute, which was superbly dealt with by home goalkeeper Fraser Forster, was their only effort of note.