Asked if he'd spoken to the Ivorian about it, Di Matteo said: "No, because I don't see any reason why I should have a conversation like that.
"Our players, I think, are fair players. Everybody has his own style but I think we are a fair team."
Drogba was accused of trying to run down the clock on Wednesday and former referee Graham Poll has warned the officials on Tuesday night may have that in mind.
Di Matteo said: "He was marked most of time by two men and I think got a lot of physical battering as well on the night. We have to be objective about our opinions as well."