Lineker spelt what he sees as the potential consequences of selling the star.
He said: "This is a defining moment where we will find out what will happen in the future for the club.
"These are really testing times for Tottenham. It's crunch time for them.
"Once you become a club that loses your best players it becomes awfully difficult to continue to compete and try to get back into the Champions League.
"If you lose someone like Modric he will be difficult to replace and for me, the club are now either going to go on to be one of the big clubs or one of the ones that are in the pack behind. If they do drop into that pack, they'll probably be there for a long time."