Liam Craig had given Saints the lead from the spot but once Graham Gartland was sent off for a professional foul seconds into the second half, the tide swung in the visitor's favour.
The former Nancy striker, Marc-Antoine Fortune, equalised in the 64th minute and notched a second from close range, either side of a Georgios Samaras drive, with substitute Paddy McCourt sealing the win with a late solo goal.
Celtic boss Tony Mowbray told Sky Sports: "Fortune works hard for the side, he brings a certain style of play in holding up the ball and bringing midfielders in. He plays his part for the team."
Mowbray had expected a difficult game, adding: "We knew they were going to make it very competitive. They are a physical side.
"We kept going. We knew in the second half we would create more chances. The sending-off was a turning point for them."
St Johnstone chief Derek McInnes thought the sending-off of Gartland was harsh.
He said: "Graham Gartland said it was accidental. I thought it was pretty harsh at the time."
McInnes felt his side had bossed the game until Gartland was dismissed.
"We were the better side in the first half, we posed Celtic problems," he said.
"I felt we had to take our medicine a wee bit in the second half."