The Tottenham man, currently on loan at Sunderland, said he had been subjected to monkey taunts long before the violent scenes which marred the end of the Euro 2013 play-off.
Rose was sent off after the final whistle for kicking a ball in anger in response to provocation as scuffles broke out involving Serbian supporters, players and other officials.
Rose told Sky Sports News: "I remember getting slapped twice and then I got ushered away.
"That's when I kicked the ball - and the referee sent me off. I don't understand, the game had finished by then but he still sent me off for kicking the ball.
"As I went off again there was monkey chanting, but the monkey chanting started long before I got sent off.
"After 60 minutes my head wasn't really on the game. They have to be banned. I don't understand how else they can learn from it, they have to be banned."