Hearts opened up the scoring in the 58th minute when Rudi Skacel hammered home from the edge of the area before Celtic's Kris Commons was shown a straight red card for a reckless challenge on Adrian Mrowiec.
Ryan Stevenson added a second against the run of play for Hearts with 10 minutes left to allow his side move up to fourth.
After the game, Celtic boss Neil Lennon admitted the clubs current position was a worrying situation and admits things will need to be done to fight back.
"It is a concern, obviously," he said.
"We have a game in hand but we have to win that game and cut it to seven and eat away at it. But we have to find more consistency in our performances.
"We have lost three games already this season and last season we lost four.
"It is a difficult position that I am in but I understand that position.
"The players are going to have to turn it around, and I am going to have to turn it around.
"A few players have to look at themselves and ask if they are doing enough at the minute.
"There are six, seven months of the season to go and we have to find consistency."