"It's been a tough three years there and obviously it was very sad and unfortunate the way it turned out in the end, getting relegated," Duff told Sky Sports News.
"That was hard to take. But I think it was best for both parties in the end for me to move on."
Duff was a talented youngster on the books of Blackburn when Roy Hodgson took in a short spell at Ewood Park.
"He made his debut, actually, before I came to the club. I think he made his debut under Ray Harford and Kenny (Dalglish) was, I think, the one who brought him over as a youngster from Ireland," he said.
"But you could see from an early age, when he was about 17, that this was a player with exceptional talent, and obviously he has gone on to play 70-odd games for Ireland and 400-odd in the Premier League."