Neil Lennon's men squandered an early lead in Russia to fall behind by a goal before a Spartak own-goal and a very late winner from Georgios Samaras allowed the Hoops to take all three points.
Gary Hooper kicked off proceedings in the 12th minute at the Luzhniki Stadium when he slotted home Mikael Lustig's low cross from six yards before Spartak responded four minutes before the break through Emmanuel Emenike.
The Nigerian striker added a second just three minutes into the second half to turn the game on its head but sanity was soon restored for Celtic when Spartak defender Juan Insaurralde was sent off for pulling back Hooper as he bore down on goal in the 63rd minute.
The Glaswegians took full advantage of the extra man but the equalising goal did come with some luck as James Forrest's 71st minute effort was saved by goalkeeper Sergey Pesjakov but deflected into the back of the net off Dmitriy Kombarov.
It set up a grandstand finish which the Hoops dominating the contest at the end as they searched for a winner which came in the final minute of regular time when Samaras nodded home Emilio Izaguirre's cross to make it four points from two outings following their scoreless home draw with Benfica a fortnight ago.
For Spartak, their Champions League campaign went from bad to worse as they gave up a 2-1 lead for the second time in a row after their 3-2 defeat at Barcelona in the Group G opener.
2012/13 UEFA Champions League - Group G