Kris Boyd equalised before Kenny Miller claimed a double - with all three strikes coming in a six-minute spell.
Rangers boss Walter Smith refused to read too much into his side's status as league leaders.
He said: "The main thing is to win the game and Rangers have to find a consistency in winning if we are to hold out any hope of being champions.
"Celtic have shown over the last few seasons that they have been able to gain that consistency. We have to show we can gain it if we hope to win the league this season.
"That's the challenge for us; it's an ongoing challenge and in this part of the season, we have been as consistent in our overall play as at any other time this season."
Meanwhile, Kilmarnock boss Jim Jefferies was disappointed at the individual errors which cost his side a possible share of the spoils.
Jefferies said: "Where did it go wrong? Three individual errors is where it went wrong.
"We put ourselves in a great position and scored a great goal and we thought the fans might start to get on their backs.
"Then we had a crazy six minutes and suddenly we are 3-1 down. Those three individual errors are what have cost us the game.
"We gifted them it - it's as simple as that."