Southgate remembers the time when he became disillusioned in the Midlands as he couldn't see where the club were heading at the time.
He told the Sunday Mercury: "It's difficult to say why (Villa can't hold on to their top players) because there will always be a club that are slightly ahead in their development.
"When I initially wanted to go it was Chelsea, obviously I ended up somewhere completely different but by then I needed a change.
"If you look at where the players have gone, it's not as if they've gone to smaller clubs.
"Villa are right up there as one of the biggest clubs. We had a team that finished fourth that was outstanding, it's still a big club now."