Chelsea have shipped a whopping 12 goals in the five matches Terry has not played in the last month.
David Luiz, in particular, has been all over the place at times, with Gary Cahill also struggling to marshall the back four.
Di Matteo said somewhat unconvincingly: "They've had some good games as well. Against Newcastle, they had a clean sheet.
"It's not just the defenders - I want to emphasise that.
"It's a team responsibility to defend."