The defender is yet to fully win over the Celtic faithful with some fans preferring to see Bobo Balde at the heart of the defence.
"Football is a game of opinions, and everyone has their own," said Caldwell.
"If it's me that takes the stick, it doesn't really bother me. I'm trying to do a job for Celtic, and if we win the league this year they can shout at me as much as they want."
While Celtic have not conceded a goal in the three games Balde has started this season, Caldwell's performances have improved since he was switched from right-back into the centre.
"Bobo came in and did his job. But it's the manager's decision. He knows the players inside out. We h ave worked with him for a long time."
"I'm pretty sure if we can put the points on the board they will not be able to come back and catch us. So it is up to us to do that.