Richards was one of the Football Association board members that voted unanimously to sack Steve McClaren on Thursday following England's exit from Euro 2008 at the qualification stage.
"It [the Premier League] has to work in the best interest of the clubs to start with," Richards told Sky Sports News.
"Everyone says there aren't enough quality players - whether there are or aren't is conjecture.
"The Premier League can't always shoulder the responsibility for the national team.
"The clubs are very successful and if you look around the clubs at the players who are there to be picked from there are a substantial amount. "I know it is difficult for the England manager to decide which players to pick - and you may say there is a problem, but going forward we are addressing that.