The Hornets were beaten 1-0 by the League One side at The Den in the fourth round but it could have been a different story if the visiting team had some luck in the first half.
"I will speak with the players face-to-face (on Monday)," said Mazzarri.
"We didn't deserve to lose. If the referee (Martin Atkinson) had given us a penalty in the second half (for a challenge on) Okaka it would have been a completely different game.
"I'm very disappointed, angry and sorry for our fans. It seemed we were never going to concede but we ended up conceding.
"A lot of players have been missing but slowly we're getting them back. This is also the reason we had to change (the team). I want a team that in the next 16 games (until the season's end) fights for every ball."