Frank was upset with fans jeering and whistling goalkeeper Guglielmo Vicario after being caught out for Harry Wilson's goal in their 2-1 home defeat to Fulham.
"It's wrong. Thomas is wrong," Cascarino said on talkSPORT.
"I've got a lot of admiration for Thomas Frank and what he did at Brentford. But you're wrong.
"You can't say they're not true Tottenham fans if they boo. They are true Tottenham fans. They're making themselves heard."
He continued: "They're unhappy of what they've witnessed. And they've been unhappy certainly in the last few weeks at the Tottenham Stadium. They have every right.
"Look booing is not nice. You can be a player that's got family or friends in the stands.
"I had it in my career. I had bad periods where you got booed. You just have to take it and you know what you can win them over.
"But when you make such an error like the keeper did. And they're booing him because they're furious with him because they're thinking, 'We've just been beaten by our arch rivals in the north London derby. Then we've been conceded five in Paris. And then, funny enough, after six minutes we're 2-0 down'. And the second one, mate, is on you.
"They're unhappy. They've been unhappy from last season. There's things that have gone on. Their results have been pretty terrible at home.
"Ange Postecoglou last year, it was a disastrous season. You know and it's going that way again. I mean because they've done okay. You know up until the last week, it has been a terrible week for Tottenham Hotspur.
"You know like Arsenal then PSG and then this game another London derby. It's been a terrible week. What do you want (the fans) to do? Clap them, applaud them? They're not going to do that."