Agbonlahor scored once and set up a double for Darren Bent in Villa's 3-2 home win over Norwich.
He said: "The good thing about us is we don't stop, we keep going, but we can't give goals away like that.
"We get a foothold in the game at 1-1 and lose a silly goal just after half-time.
"The ball should have gone into row Z of the stand. It puts you on the back foot and then you are down to 3-1.
"We came to try and win and the first goal was a world-class goal. But Villa are a very good side especially here with the crowd they've got.
"We got a lifeline with the second goal (from Steve Morison) but you can't give away goals like that every week and expect to keep bouncing back."