Jonathan Barnett, however, says Bale would be happy to return.
The relationship between Zidane and Bale was basically non-existent by the time the Welshman left for Spurs last summer, and Barnett feels that the French coach doesn't love and appreciate the player as he should.
"He still loves Madrid," Barnett told Goal.com.
"He hasn't got any problems with the club; it wouldn't be a problem for him to return.
"They just need to decide if they want him back, I guess you'll have to ask Mr. Zidane if he wants him, but I don't think so."
Real Madrid supporters grew frustrated with Bale near the end of his time with the club, blaming him for a lot of the team's on-pitch struggles, but Barnett believes that the 31-year-old doesn't need to pander to them should he return.
"He doesn't have to do anything, he does his talking on the pitch," Bale's agent added.
"He is a footballer, that is his responsibility, it's not his job to be friends with the fans.
"He did everything correctly on the pitch and was incredibly good for the club, he never said anything and he won a lot of trophies, several of the great Real Madrid goals are from Gareth, so I think when people look back they will realise how idiotic they were with him."
When pressed on whether the relationship between Bale and Mourinho is broken, Barnett suggested not, but couldn't resist a slight dig at the Portuguese coach and his Tottenham team.
"They have a good relationship, there is no problem," he confirmed.
"It's journalists who create the problem, but there is no problem.
"We expected him to play more, things haven't gone that well, but the team haven't done that well either."