Craig Levein’s side were beaten 2-0 by Belgium in Brussels to fall to the bottom of the Group A qualifying table, eight points behind the top two teams.
Talk surrounding Levein’s tenure as boss has increased since the loss but Caldwell is of belief that the team is not travelling too badly despite two home draws and two losses on the road to begin their campaign.
“We know we have let ourselves down. We have to take responsibility for that over the four games so far,” said the Wigan Athletic centre-back.
“We have not hit the heights we feel we can since the campaign started.
“As players that is our responsibility and we need to look at ourselves individually and as a team and admit it has not been good enough.
“You can see it is the case the players are all behind the manager.
“When a manager gets sacked usually a team is in disarray. I know I have been part of teams that have been like that.
“I don’t think that team tonight is one that was in disarray.”