Croatian international Modric is poised to complete a move to Real Madrid in the next 48 hours.
And Defoe, who scored England's winner against Italy in Berne last night, said: "A lot has gone on but as players we don't know what's gone on.
"Luka is a fantastic player. He's been good for the club to be honest.
"If he does end up going, for us it is hard to lose someone like that.
"But, for him, I would say good luck to him because he's a fantastic player."