The League One side put up a stubborn fight against David Moyes' side.
But Pablo Zabaleta eventually found the breakthrough after 74 minutes, with Pablo Fornals adding a second deep into stoppage-time.
Speaking after the game, Moyes said: "We're really thrilled. These games have caused us difficulty over the years and Gillingham nearly did that again tonight - they were really good, really competitive, hard to play against but we had a it of quality.
"At half-time I was more disappointed that we had not tried to play enough and not got the ball down enough. The defenders were slashing at clearances and did not show enough composure but it was always going to be this type of game.
"Great credit to the players and they stuck at it and showed great resilience.
"I thought Steve Evans' team did very well and they will think they were a bit unlucky but I thought we could score again on the break."