Caldwell was among the sales from Parkhead during the winter market.
"The transfer window was the turning point for Tony," Caldwell told the Sunday Herald. "To let go of so many first-team players in such a short space of time was always going to be a risk. We're not talking about squad players. Of the seven who left Celtic, four or five would have always been in the side.
"Myself, Danny Fox, Barry Robson and Scott McDonald all featured in that draw with Rangers. Big Mick [McManus] was on the bench. We dominated Rangers. It was as one-sided a derby as I've ever seen, but within weeks we had all left the club.
"To lose almost half a team in one transfer window is always going to give you a problem. Tony probably thought he was getting quality players in but how often does a new team click straight away?
"In time, all of these new players might gel, but there was a title to win. When Tony made the changes, Rangers were ahead and you don't get time to gel a team together at Celtic. Old Firm football doesn't work that way."