The League Two side, currently 64 places lower than the Gunners, triumphed 3-2 on penalties after the match ended 1-1 after extra time at Valley Parade.
Midfielder Gary Jones, a scorer in the shoot-out, called it "one of the best nights of my life" before adding: "I can't believe we've just knocked Arsenal out of the Capital One Cup. It was 11 v 11 and we had to keep our shape and frustrate them."
Goalkeeper Matt Duke saved a penalty from Santi Cazorla and said: "We set our stall out and the men in front of me were fantastic. We are really fit as a team and that showed."