Cole led England for last night's draw with the Republic of Ireland.
Wilkins told TalkSport: "I'm sure he's felt like turning it in because of the adverse publicity.
"It got to him to that degree. I remember one conversation where I said, 'This is ridiculous Ashley, you've got so much to give. Don't be doing that [packing in], that's daft'."