The Sky Blues held a 2-1 advantage heading to Germany on Wednesday, but it was reduced early on when Jude Bellingham curled in a superb strike from the edge of the box.
The goal gave Dortmund the aggregate lead on away goals and they held on for as long as they could until City finally got the breakthrough.
Emre Can headed the ball into his own arm, and after consultation with VAR, the referee awarded a spot-kick to City.
Riyad Mahrez stepped up to fire home past the goalkeeper's left to put City ahead in the tie.
There was a chance for Dortmund to square things up, but Phil Foden sealed City's passage with a fierce drive with 15 minutes remaining.
The goal sparked jubiant celebrations with the City players and manager Pep Guardiola, perhaps relieved knowing they are one step closer to shaking the tag of perennial Champions League losers.