The 25-year old Manchester City keeper has come under criticism recently following an almost flawless campaign as he helped guide his side to the Premier League title last season.
Hart has made a few errors in recent times but Southall feels the negativity toward the England number one is unwarranted and has backed him to become a force in the world game.
"I just think it's a load of pants, really," Southall told Goal.com of the criticism.
"If you look at him, he's the most improved goalkeeper I've seen for a long, long time.
"He's gone in there and now he's let in a couple of goals that people think he should not have let in. That was always going to happen."
He added: "Best in Europe? Not at the moment because he's still a bit young.
"He hasn't learnt what he needs to learn and that takes time, I'm sure he's learning his craft every day.
"In time, if he carries on going the way he is, he's going to be one of the best in the world. He's probably already in the top 20 at the moment.
"Now he's just got to work himself up to the number one position. I'm sure that's his aim and his ambition."