England keeper Hart rushed off his line in a bid to punch clear a 71st-minute corner.
But Poland's Kamil Glik got there first to head home.
Hart admitted: "I came and didn't get there and the lad got a good header. It's my fault. I probably should have punched it or stayed in my goal.
"I didn't do either and they scored. I'm gutted that it has cost us three points."