Boyd had only just come onto the pitch before he was firing ten man Rangers to a most unlikely three points.
Kevin Thompson had already been show a rd card after only 14 minutes and David Witteveen went on to fire Hearts into a half-time lead.
Lee McCulloch restored parity to set up Boyd for the dramatic climax.
Gers boss Walter Smith had no doubts Boyd would hold his nerve from 12 yards.
He said: "Kenny Miller had taken a bad knock on his arm in the first half and had to play most of the game up there on his own.
"Boyd then came on and had a good touch for Steven Naismith. When you get a penalty, you are always confident that he will score it.
"It was a good finish from him and then we had to hold on a little bit at the end, which you would expect to do away from home and down to 10 men."
Smith believes the punishment handed out to Thomson was harsh.
He said: "I thought he gave the opportunity to the referee to order him off. Maybe where he was it looked worse but I didn't think there was any great contact with the player.
"From the dug-out area you could see that and the player carried on playing for the rest of the game so I don't think there was great contact.
"My own view was that there was a similar challenge beforehand and the referee allowed that one to go with a yellow card and I thought he might have done the same with Kevin Thomson."
Smith added, "Hearts are a fairly direct team, they are a strong team in the physical sense, and we've been caught slow to start here on a couple of occasions and that was one of them today.
"We tried to play a little too much in the first five minutes or so and never really got started. But I'm obviously delighted to get the three points."
Hearts manager Csaba Laszlo refused to comment on the penalty decision.
He said: "What I think at the moment is not so important - you must decide for yourselves if it was a penalty or not.
"I don't talk about the referee's performance. If we speak negatively about it, it doesn't change the situation. We have still lost the game 2-1."
Hearts had suffered a 4-0 thrashing against Dinamo Zagreb in the Europa League on Thursday.
Laszlo added: "It was possible to take a bit of confidence back after the Zagreb game, but this was painful, this was painful for me.
"This was a game that was possible to win and we had good chances in the first half. But by the end of the second half you could see the tiredness after Zagreb.
"This was our third game in a week and for a young team that's not easy to handle."