The Hammers conceded a penalty which was converted by Christian Benteke in the 74th minute before Charles N'Zogbia produced a stunning free-kick four minutes later to give the home side a nice advantage going into the final ten minutes.
Ashley Westwood's own-goal three minutes from time gave West Ham some hope before Villa keeper Brad Guzan pulled off two late saves to deny Kevin Nolan and Carlton Cole leaving Allardyce to rue what may have been.
"I thought we had saved it but we should never have been in that position. That's the biggest disappointment," he told Sky Sports.
"We should never have got ourselves in a losing position because we'd created the chances. We can't play any better than we played.
"We then created the chances to get ourselves back to 2-2 and we failed to do so and that's been our problem away from home this season.
"It was great goalkeeping in the end. Guzan pulled off some really good saves.
"I thought we sat comfortable from the chances Villa created and we gifted them a soft penalty (when Mark Noble brought down N'Zogbia) when we should not have been in that position.
"It was a great free kick from Charles N'Zogbia so we can't criticise that too much. The overall situation was disappointing because of the way we played.
"We dominated and need to put the ball in the back of the net when we get a chance."