The Swans are beginning the hunt for a new manager after agreeing a compensation deal with Liverpool to allow Brendan Rodgers and backroom staff members Colin Pascoe, Chris Davies and Glen Driscoll to move to Anfield.
Whelan told Sky Sports News: "I had half a dozen phone calls saying Swansea were wanting permission to talk to Graeme and I said I really don't know because our chief executive (Jonathan Jackson) was in a Premier League meeting.
"But he rang me at 2.30 and said that was the case and we gave them permission immediately.
"Roberto doesn't believe in holding people back. It's a free country but if he goes we'll miss him immensely.
"He's ambitious, has played for us and is a football man through and through.
"He's worked with Roberto for a few years but I think he's up for the challenge and is a very good lad.
"Roberto would like him to stay - they've been a great partnership for quite a while and the first thing Roberto said was he'd miss him immensely.
"But you can't stop progress and it's unfair to say 'you can't take that job, you're under contract'. It's not fair."