Inter won the Italia derby 1-0 thanks to a confusing goal from Sulley Muntari, who appeared to be completely ignored by Manninger.
"I was worried because I saw the shot from Ibrahimovic late, as there were defenders in front of me," said the Austria international.
"I saw it was going well wide and was relieved, but Muntari came out of nowhere for that tap-in and we all thought he was offside."
That proved the decisive moment in the Derby d'Italia, as Inter ran out 1-0 winners, ending the Bianconeri's run of seven consecutive victories.
"It is not expected for any team to come to San Siro and beat Inter, but until the goal it was very balanced and could easily have ended with a draw.