Terry has been banned for four games and fined £220,000 by the FA for his rant during a match last October.
Boss Di Matteo said: "Certainly the image of the club has suffered from this issue. There's no denying that.
"It's put a bit of a cloud over the image of the club. It's not for me to judge the whole package. I'm the manager, not a judge of people's lives.
"I judge on what I see on the training ground and the pitch. That's what my job has to be about."
Di Matteo added: "I think he realises he let himself down and the club in that day and moment.
"It's a shame that we can't talk about football, the main priority. I understand you have to report current news. I've got a big week coming up, one of the most important weeks of the season and I don't get one question about football.
"John's been a great servant over 555 games. He's made a mistake, let himself and the club down. He's done a lot of good work as well. I do not regret supporting him.
"I believe the evidence he gave to the magistrates court is exactly what happened on the day. He was cleared in court and we take that as a result of it.
"He realises his language was not appropriate. He realises he let himself, the club and the supporters down and will get punished for that."