The Hammers came from a goal down to take a 2-1 lead into injury-time at King Power Stadium before the Foxes managed to grab a point when Leonardo Ulloa converted from the penalty spot after Jeff Schlupp had been felled just inside the area.
In the wake of the 2-2 draw, Bilic believed the East Londoners were good value for all three points in what was a dramatic clash in the East Midlands.
"I thought - and I still think - we did enough to win the game. But then at the end we first made a mistake because we had the ball in the middle of the park in a situation where you know they are going to take risks," he said.
"They don't care about if they are going to concede one more or not.
"But then you can't lose the ball in that situation. After that what happened makes us and me feel very frustrated, very disappointed."
Andy Carroll had converted from the penalty spot before Aaron Cresswell fired home a stunning volley as the Hammers fought back after conceding to Jamie Vardy's fine finish in the 18th minute.