Hart has been hammered after City's defeat at Cardiff City on Sunday.
But Southall told talkSPORT: "Hart is still five or six years off his peak. He's not going to be perfect, is he? So I don't get where all the criticism comes from.
"Yeah, you play for England so you're open to more stick than normal I suppose, but they've got to remember there's nobody better than him. He's as good as anything else in the Premier League.
"I always work out that if you're good, you're always going to get criticised. And if you're rubbish no-one gives a monkey's!"
For the big stories and the banter follow us on Twitter: @tribalfootball