Spurs were in complete control of their Group J game against the Greek side and took the lead after 35 minutes when Michael Dawson glanced Tom Huddlestone's cross past Orestis Karnezis.
Panathinaikos took full advantage of Spurs' missed chances through striker Toche, who left Dawson for dead and turned in a Giorgos Seitaridis 13 minutes from time.
"I don't think we were complacent," Villas-Boas said.
"We have to give credit to Panathinaikos, they pressed much more and started to gain belief from that and that led to the goal.
"I think the whole team felt a bit disappointed because the first half was so positive and we were in control of situation.
"We were a bit sloppy in the second half. I understand the disappointment at not putting the game to bed, but in the end we congratulated the players because if it had been other way round, the whole perspective would have been different.