Abramovich landed by helicopter and drove the few hundred yards to the stadium in a bulletproof limo driven up from London.
He spoke just six words to his host, brought his own drinking water and, to add insult to injury, refused a famous Wigan pie.
Whelan said: "Abramovich had just bought Chelsea and he used to travel then. He came to Wigan in a helicopter and sent three men up on the Wednesday and two more on the Friday.
"I let him land about 300 yards from the ground. I said I would put a car on to pick him up but he had his own sent up from London. Bulletproof.
"It picked him up and brought him to the stadium. He stood in the boardroom and wouldn't move out of the corner. He wouldn't have anything at all to eat or drink.
"I offered him a Wigan pie. He wouldn't have it - he probably thought it had been poisoned.
"He had nothing, not even a glass of water. He brought his own water with him. So he just stood there in the corner.
"He went out to watch the match with six bodyguards around him. Afterwards he came back into the boardroom and said, 'How long before I can go?'
"I told him he would have to give it ten or 15 minutes to let the crowd go. Then he was back into the bulletproof car and off.
"Now, whenever I go to Chelsea I never see him. He never comes to say hello. Never.
"I do wonder sometimes, 'We said hello to you at Wigan and offered you a pie. You should come and talk to us. Wish us luck now and again when we go to Chelsea'. It's strange when you go to a club and don't see the chairman."