The 29-year-old playmaker, who starred for France at Euro2016, is a target for Spanish giants Real Madrid who would have to pay an exorbitant amount to lure him from East London.
Sullivan is aware there will be interest in Payet for the remainder of the summer transfer window but is not concerned that the former Marseille will up and leave.
"Am I worried about losing him? No, because he's under contract," Sullivan told the Daily Mirror.
"There might be six weeks of silliness when he bangs the table. But just because a big club wants him, tough luck.
"He signed a contract and you take the rough and smooth of a contract.
"Sometimes clubs have got to be tough and stand up and say 'he's contracted to us, he's our player'.
"Just because he can get more money somewhere else he expects us to sell him.
"Well, we're not selling him unless someone wants to break the world transfer record and then we might consider it!
"Even at his age he must be worth £100m."