James believes Hart's inconsistency is down to the lack of decent competition for the Three Lions' number one spot.
He told talkSPORT: "I think [criticism] is expected and that's not because of the performances necessarily. It's because he's now the number one, he's played at a tournament and he's won the Premier League as well.
"All of a sudden everyone is looking at him. Every little mistake that's made will be highlighted and I think as well the problem that he's got, in a strange kind of way, is that he's not really got the competition and therefore he is essentially the flagbearer.
"Rob Green is around [at QPR] as a number two now. You've got Jack Butland who is playing in the lower end of the Championship. So the English goalkeepers coming through aren't really challenging him."