The Saints travel to West Brom on Monday night still searching for their first Barclays Premier League win of the season.
"It comes and goes, speculation," he said.
"Whatever team is in the bottom three their manager will be under pressure, or at least the media will think that.
"There has been speculation since the start of the season about who was going to be the manager and ifs and buts, as a player you just get on with it and do as best as you can, not just for yourself and the team but for the manager as well.
"He (Adkins) got back-to-back promotions so has got to be given great credit for what he has done already and we all respect that."