Spurs went ahead in the first-half with a diving header from Harry Kane, assisted by an inch-perfect cross from Sergio Reguilon.
Cavaliero rose brilliantly and punished Tottenham for not killing off the game.
It means Spurs have won just two of their last eight league games, leaving their title bid in serious doubt.
Speaking after the game, Tottenham boss Jose Mourinho was frustrated with the performance.
'In the first half Alphonse Areola made some impossible saves, couple of others in a second, too.
"We have to kill a game and we didn't - but you have to keep a clean sheet, not make mistakes, so it was a very avoidable goal. The markers are there, there wasn't even an advantage in terms of numbers.
"They were intelligent enough to understand the way they play, they change, they become more defensive and they are getting results. I thought they were a bit lucky but they were good.
"We have bad results and we should - and we could have - avoided these results."