Potter and his extended backroom team were axed by the Hammers this morning.
"The timing is unusual two days before a game but the writing was on the wall for Graham," Cole told TNT Sports.
"To lose three London derbies the way they did wasn't good enough. I don't think he was given enough backing, in this transfer window in particular.
"They should have gone, 'If you're our guy, there you go, what's your plan?'. But I think they dithered, thought 'Is he or isn't he (the guy)?'.
"Ultimately it was going in the wrong direction so I understand why the club did it.
"The real shame for Graham is he came in at a difficult time for the club. I would imagine the conversations were 'We're going to restructure, change the style of play, bring in players you want to bring in' but that never happened.
"I wish him the best in the future and hopefully things can turn around for West Ham as well.
"If I was in charge of West Ham, I’d rip up the whole club, invest in the academy, and get the right type of player in for the manager."