Spurs were hammered by their arch rivals after going a goal up through Emmanuel Adebayor who was then sent off in the 18th minute for a reckless challenge on Santi Cazorla.
It was the type of loss that could wreck the mindset of a group of players but Villas-Boas feels the only thing that needs fixing is the defensive side of their game.
"When you are poor, it can do damage, but when you play like this I don't think it can," he said.
"We are a good attacking side but we have been conceding a couple of goals," Villas-Boas said. "We understand that we have to do better.
"But we suffered three in the first half, we suffered two in the second half, with one player less.
"There was attacking volume for both sides. Arsenal were efficient in their chances and we weren't as efficient in ours."