Hazard, desperate to move to Real Madrid, fears the Blues will hold on to him if their two-window transfer ban is upheld by the Court of Arbitration for Sport.
Di Matteo, who was Blues boss when they signed Hazard from Lille in 2012, told The Sun: “Who knows about Hazard? Chelsea have a transfer ban so it is going to be very difficult for anybody to leave that they don't want to leave.
“He's the player that makes the difference. He can change a game individually and there aren't many players like that. It would be a real shame to lose a player like that.
“Losing Hazard is not great but I'm not sure it will be that easy for him to leave.
“The club has a transfer ban and letting a top player leave and not being able to replace him, it would be very tough for a top club to buy him.
“Nobody is bigger than the club. It's understandable that he may want to experience a different league after being in London for seven years but it might not be this summer."