Guiseley boss Mark Bower, who played with Vardy at Halifax, says the England star's bad-boy reputation scuppered a move to the Bantams.
He told The Sun: “They didn't want to take it any further. Their loss I guess.
“The lads at Halifax knew he would go on to bigger things — but we didn't think he'd be playing for England."
Oldham also wanted Vardy but could not afford to take him from The Shay.
Paul Dickov, then Latics boss, said: “If you look at Jamie now, it was probably a good thing it didn't happen!"