The Blues released the centre-back after making over 700 appearances for the club.
"I didn't want to play in the Prem, that was a big thing for me, I didn't want to play against Chelsea," Terry told Sky Sports.
"But obviously I gave Clement the service because I felt I owed that to him with what he done for me, how he helped me growing up. 'But we met and having not wanted to play against Chelsea he pushed me really close as the way he dealt with me. But in the end I couldn't do it."
The 37-year-old eventually joined Championship club Aston Villa, who are currently fifth on the table, ten points behind an automatic promotion spot.
Asked if he will have to quit if they get promoted, Terry replied sarcastically: "Maybe, maybe, we'll see."
He added: "First things first, I want to get us up, that was my big thing. We're going okay at the minute, obviously a setback at the weekend losing to Derby [which they lost 2-0]. But listen, so many games and they come thick and fast in the Championship as well."