United badly missed injured skipper Chris Morgan, who has been ruled out for the season with a cruciate knee ligament injury, and the makeshift United defence were at fault for both goals.
Speed said: "It's obviously disappointing, frustrating. Two sloppy goals have cost us.
"With the situation at the moment, the form we're in and the run of results we're in then when you go a goal behind, you give yourself a mountain to climb. They got themselves back into it through their workrate, which we always have, but then they've given it away again with another soft goal.
"You can't keep giving goals away like that and expect to win games."
When asked about Keane's comments about Ipswich "riding their luck," Speed replied: "I think when things happen for you, you've got to take advantage of it and they did that.
"I'm fed up of coming in and saying things aren't going for us but it's more than that. We've got to make our own luck."