The French midfielder was originally recruited by United from Le Havre back in 2009.
He would last just three years at Old Trafford before joining Juventus in acrimonious fashion, after which Ferguson accused him of 'disrespecting' the club.
Pogba returned to the Red Devils for a club record deal worth £89.3m in 2016, but his inconsistency and attitude issues have riled supporters in the time since.
Pogba replacing Paul Scholes in 2012
The 27-year-old exacerbated the issue last month whilst on international duty with France, stating it was a 'breath of fresh air' being away from Old Trafford.
And it seems Ole Gunnar Solskjaer has lost all trust in the World Cup winner, having started him in just five matches this season.
With Juventus and Real Madrid linked with Pogba, Brazil thinks it would be best for both parties to move on.
"Fergie sussed Pogba out from an early age," Brazil told Tribalfootball. "He took him to the airport and basically said there's the plane, see you son.
"And I think Ole would be better off without him at the club. People still tell me you can't, he's the best player at the club.
"If he is, I haven't see it, or at least, I haven't seen it consistently. And he keeps talking about when he is away with France or whether he's on holiday.
"I think he has disrespected the football club of Manchester United, I really do. Sir Matt Busby would never have put up with them, Fergie certainly didn't.
"And I think United have made a big mistake here I really do. Don't get me wrong, what he's achieved in the game, World Cup's, winner in Italy, there's no doubt he's got amazing talent but he doesn't do it consistent enough.
"And sometimes you think about care. I hope he cares. But sometimes it looks as if he doesn't. He won't be for me. He'd be one I'd move on definitely."
Pogba on duty with France last month
Pogba's comments while with France highlight a key problem that Solskjaer has been trying to root out of his United squad.
And according to Brazil, it is imperative that Solskjaer continues to create a positive environment in the dressing room.
"You need harmony in any club. Klopp has got it at Liverpool. Fergie had it at United where he didn't have a problem if he left some stars out. They accepted they had to fight their way back in. That's what Manchester United have to get back too.
"Sometimes I ask myself, do those players care that they are not playing or not, are they happy to sit on the bench and draw massive salaries, the game has changed so much.
"Pogba, I'd love him to go away, to Paris or wherever, and show us how good of a player he is every week. Because he has clearly got it but he hasn't shown it yet."