The Nerazzurri were trailing 2-1 until the 88th minute, when Fernando Couto was sent off and conceded a penalty for a dubious handling offence.
Zlatan Ibrahimovic converted the spotkick before swooping for the winner three minutes into injury-time.
"I think the ball hits Fernando's head and then on to his arm. It could have gone anywhere, but it did hit his arm. It's unfortunate that it cost him a penalty and red card, but that's the sport.
"We played arrogantly, Parma were forcing us to run in circles and Parma deserved far more. We were struggling on the flanks and crosses kept coming in, so I had to try to change things. We were predictable in every element."
Douglas Maicon, Luis Figo, Dejan Stankovic, Walter Samuel and Olivier Dacourt were sidelined, but Nuno Maniche received his less than impressive debut.
"Luis Jimenez didn't have much space and should have spread the play out wide, but that is true of everyone tonight. I think we also paid the price for injuries and it wasn't easy to give Maniche his debut. He had problems today.
"I think Maniche needs time to settle and get to know his new teammates. He hadn't played in a month, so it was difficult for him."
Mancini is now the Inter boss with the fifth most matches under his belt, as this was his 200th official game.
"I hoped I would reach this achievement and to spend a lot of time on the Inter bench, but one always does when starting out."
He also said: "We didn't put in a great performance and Parma comfortably deserved a draw. We did not play well or take the right approach, as we underestimated our opponents and it was not easy to get back into the game."