Two goals in the final five minutes gave Carlos Carvalhal a dramatic debut victory as Swansea boss.
While Silva declined to refer to Andre Gray by name, it was clear it was the record signing's miss seconds before Jordan Ayew's equaliser which irked him the most about how the Hornets had failed to put the game beyond Swansea, and add to Andre Carrillo's 11th-minute opener.
He said: "It's our job to find out why we cannot kill games off. It's clear, we have to put the ball in the net.
"We made enough chances, the game was under control and the opponent didn't create chances to put us in danger until then. We needed to score a second goal. We needed to be assertive."
Silva continued: "I'm very disappointed for us. I'm disappointed for our fans as well.
"We did enough to win the game, our opponent had one shot on target from Ayew, but after that they didn't create a chance until they equalised.
"Heurelho Gomes didn't make a save until then, and we had enough chances to win the game and kill it off.
"We had a clear chance just before then, and at this level when you don't score it gives the opponent some belief.
"We didn't create a big number but there were enough. They won a second ball, and even after that we needed to keep focused.
"It's a big mistake, our players didn't stay focused until the end and we were not in position."