Sherwood was shown the door following the Villans' loss to Swansea in the Premier League on the weekend, but Richards believes the decision may have been made too soon.
"Tim was unbelievable for me. I think he has got me back to my best," Richards said.
"I feel comfortable. He gave me the captain's armband, which was a massive confidence boost. Tim was great for me but we know that, in football, if you are not getting results, clubs look to make changes.
"I think Leicester was a turning point in our season. When you are playing outstanding football and go 2-0 up, and then find yourselves 3-2 down at the end of the game, I think that knocked a bit of confidence out of the lads.
"For 70 minutes it was not like the Aston Villa of the last two years. They did not have the free-flowing football that Tim brought here.
"I wish he had a little bit more time, because I think he was on to a really good thing. But when you are not winning, decisions like that are going to get made."