Newcastle were leading 2-1 at the Bayer Arena before Alejandro Grimaldo found an 88th minute equaliser.
Asked if defensive lapses are hurting them, a disappointed Howe said afterwards: "I can't disagree. I think we have work to do.
"But I think we have work to do in every part of the game. I think we are team that are building in the right direction.
"We have had a difficult summer. We saw that in our early performances. The last six have been a lot better. But we are not perfect, and we are doing our best to develop and grow.
"There are positive signs. But there are still some moments where we shoot ourselves in the foot to a degree, and that was there again last night. I know it is a frustration. Not just for me, for the players and for everybody. We are costing ourselves at the moment."
He added, "Rather than focus on the lateness of the goal, it is the way we conceded. The timing for me is irrelevant.
"The first one in particular that got me because we have worked a lot on set-plays in between the last game but we did not defend it well enough.
"I thought we did well to respond to that. We'd done the hard work, we were in front so it is a case of having to defend well."