Passions were, perhaps understandably, running high after a derby defeat by arch-rivals Newcastle in the second game of the campaign, but Gardner insists the reaction has been over the top.
He told The Journal: "We are four games into the [league] season - it's ridiculous.
"People are writing us off, but there's no way you can. I think people are jumping on the bandwagon.
"We haven't played that badly, we are not struggling. We are four games into the season, we have got two points and we have played Newcastle and Chelsea at home. We have actually had two good results away from home.
"I don't understand why people are saying we are struggling because at the end of the day, we are not. We are playing well, we are just lacking a couple of goals, which will come.
"We are a top-quality team and there are four games gone. I can't believe it. The players we have brought in - look at them.
"There's experience, there's youth, there's pace, there's creativity. There's no way we are going to struggle at the bottom all season."