Early goals from Harry Wilson and a brilliant Alex Iwobi strike gave the Cottagers a 2-0 lead at halftime.
However, the visiting Londoners reduced the deficit through Richarlison but the Cottagers held their nerves to secure victory.
“I believe the three points clearly we deserved,” Silva said.
“In terms of the score, it should be different with the amount of chances that we created.
“Of course, first half, the beginning of the game, we were clinical, ruthless. We created and we punished them.
“But the number of moments that we had second half to score, clear chances to score the third goal and kill completely the game, and we didn't. That is the reality.
“And you know, anytime at the level that we are playing, with the individual quality on the other side, they can punish you, which is what Richarlison did in that moment.”