He returns to St James' Park tomorrow as West Ham boss insisting there is no animosity.
But there is still plenty of frustration for him and his missus, Lynne, more than four years later.
Allardyce said: "It was the hardest sacking to take because I thought that job was THE right one.
"I thought it was the next step, a big club, big fan base, big budgets. And unfortunately I never saw any of that.
"The issue is your professionalism has been questioned, your ability to do this particular job has been questioned.
"It's the damage to your career that you've spent years and years and years putting together. The only reason cited to me was they wanted to move in their own direction because I wasn't the man that was selected by them, I was already there.
"At the end of the day you have to accept that and move on.
"I felt terribly low. My wife said 'for God's sake you're better off out of it, is it worth it?'
"They go through more stress and pressure than you because they sit and watch you go through it all and see what no one else sees. That's why she said what she did.
"But she also understands I am a football man and even in the tough times I need to do this job, it's a drug.
"I shook hands with Mike when I went back with Blackburn.
"He has worked it out - how to run a football club - and he is getting Newcastle to be what I wanted them to be, a team pushing for Europe and the Champions League."