Villa were beaten after Steve Sidwell opened the scoring in the 21st minute before Dimitar Berbatov scored from the penalty spot on the half-hour mark after Alex Kacaniklic collided with Leandro Bacuna in the area.
Lambert felt his side were hardly done by with the penalty, especially when Gabriel Agbonlahor was felled by Aaron Hughes late in the game, and rued the fact they were unable to get at least something from the trip to southwest London.
"I don't think we played as well as what we have been but I still think that some of the decisions went against us," Lambert told Sky Sports.
"I think the referee's performance for the whole game was poor. I think some of his decisions were poor."
He added: "As I said, I thought the referee's performance today was poor. There's no point in going on about it."
For breaking stories and all the great banter like us on Facebook: facebook.com/tribalfootball