The 25-year-old was a £9million signing from Wigan in the summer but has struggled to replicate his Latics form in the Midlands.
Grant told the club's official website: "You know what football's like, people talk about somebody moving every day. I think it was Benty last week, now it's Charles. I don't know who it will be next!
"People are talking about players going here, there and everywhere. Sunderland must have a few bob left if they are going to try and buy Charles!"
"Listen, top players are always going to be linked with other clubs, but if players leave here they should only be going to top four clubs because there's not many clubs bigger than this one," he added.
"In terms of the stadium, the history and the fans this is a big club and what we want to do is make it somewhere that everybody wants to come and play and that the boys here want to stay at."
And Grant is convinced it's only a matter of time before the former Newcastle and Wigan star regains his form.
"We feel in the last few weeks that Charles is starting to get there," he continued.
"The boys appreciate his talent because they see it on the training ground. That's why we're confident that there's a lot more to come from him."