Varney says the only thing that can rock Pool is the loss of key players and their manager.
Captain Charlie Adam has been linked with Bolton and maverick boss Holloway with the Blackburn job.
The frontman admitted Olly's departure would make him think twice about making his loan move from Derby County permanent in the summer.
He said: "It's so important we keep this group together. If the gaffer left it would be catastrophic."
Blackpool>It's all kicking off again! Click here for the No1 coverage of Blackpool's winter market transfer news.