Now as the Liverpool midfielder closes in a third Premier League winner's medal, his former boss reveals he couldn't be more delighted.
“At the end of the day he was consistently excellent for us. We were doubling our money but he was irreplaceable for me.
“James was going there to further his career and certainly further his future financially. But to me, we were trying to compete with the likes of City and Tottenham at that particular point in time. Maybe another season and we might have done it. It was disappointing.
“But I've got to say, I could not be more pleased more for him for getting the career and the medals he so thoroughly deserves."
He added: “He can run with the ball, score you a goal. He was terrific. He was never my teacher's pet, but he was terrific for me.
“Truth be known I actually had a different teacher's pet every week, but I'm glad to know that Teacher's Pet went on to win so much in the game while I would politely suggest those who said it, the likes of Steve Sidwell and Curtis Davies haven't gone on to have careers anything like the one that Milner has enjoyed.
“James just covered so much ground. He is like the identikit pro. He doesn't drink. He puts everything into his training and his game. He's put football at the top of his list of priorities so it's certainly no coincidence he's still going strong.
“I tried to get him first and on deadline day he was less than three miles from Villa's training ground and he was told to go back. He would have been very cheap then – about £4m, if I recall. I chased him again and kept going and when things weren't working out for him at Newcastle we paid £12m for him. Not bad business."