The Wales winger Bale was booked for simulation for the second game in a row in Saturday's 3-0 win at Fulham.
"This debate really annoys me," Ginola told BBC Radio Wales.
"The kid is doing so well. He shouldn't carry this on his shoulder. I don't understand why we put pressure on him with false ideas. It's unfair."
Ginola believes officials should protect players of Bale's ability rather than punish them and said it was hard for a player to shake off a reputation for diving once he had been singled out for criticism.
"You don't need a big tackle to go down," said Ginola. "A little push would be enough to lose the balance.
"Football is about entertainment. When you have players like that who entertain you week in, week out, you have to protect that. It's so precious.
"You have to take care of them because they are little diamonds in a league like the English Premier League.
"I thought he was a diamond when I saw him play against Inter Milan and score three goals [in 2010]. He is special."