Milner's sending off came about in the 23rd minute when he committed a professional foul on Long as the last defender with the Republic of Ireland international about to latch onto James Morrison's through ball.
It took the Baggies just under 45 minutes to capitalise on the man advantage and it was Long who did the damage when he slotted home past Joe Hart after receiving the ball from substitute Peter Odemwingie.
But it was City's sub Dzeko who was the man of the moment, equalising within a minute of coming on, courtesy of a header from a Carlos Tevez free-kick before he snatched the victory in the final minute when he steered home Sergio Aguero's neat pass.
Moments earlier, Hart had come up with a vital save to deny an acrobatic Romelu Lukaku effort but in the end it was a deserved win for City, who despite having one man less for the majority, forced the issue much more than Albion.
The victory allows Roberto Mancini's side to stay third on the table with 18 points from eight outings as Steve Clarke's Baggies dropped to seventh with their first defeat at home this campaign.
The Hawthorns, West Bromwich - 24,891