Chelsea boss Di Matteo faces his old club today.
Brunt told The Sun: "At the time when Robbie was sacked, things weren't going very well here.
"It's all on the manager's head these days but the players had to take some of the blame as well. It was a confidence thing, we got stuck in a rut.
'The club made a decision - and looking back now, it was the right one. If Robbie looks back on it now, even he would say it was the right thing for him - look how well it's panned out.
"All credit to him for what he's done since.
"But sometimes you have to accept that things don't work out at a club.
"After winning the Championship, the Premier League was a different step - but then he went to Chelsea and with the squad they've got he maybe got them doing some of the things the previous manager couldn't.
"If you look at the big picture, things have worked out very well for both clubs.
"We're getting a decent link with Chelsea now under the new manager. But we've moved forward in the last few years and Robbie played a big part in that."