Ahead of tomorrow's clash at Wigan Athletic, Ferguson said: "Sometimes I look back at the time and it does not sit easy with me.
"I don't know how I would feel now watching this sitting in the directors' box. It would have been difficult, this is a fantastic team."
Fergie added: "The decision was taken because my wife told me I was not resigning. It's the only time I have been bullied!"
Some dared to question his decision as the club went three years without winning a title.
But Ferguson knew what he was doing and recreated a team not just for the here and now, but the future.
He said: "The expectations to deliver here are so high. It was not easy because we had to let players go who were part of the family here. Players like Roy Keane and Phil Neville.
"It was a period of change. But the manager is the only one who has the patience. The supporters don't have it, only you have the patience."