Carrick was appointed on an interim basis in January and signed a two-year contract after the club secured a Champions League finish in the Premier League last year, despite a disastrous start under former boss Ruben Amorim.
United are now in the process of finalising a deal to bring in Mauricio Pochettino as his replacement, a deal which would mean the Red Devils shelling out £7M to secure his services.
Carrick had been in the job for 10 months, and the club is now in a revolving-door cycle of coaches following the departure of Sir Alex Ferguson way back in 2013.
United took only eight points from 10 Premier League games and sat in the relegation zone and had won only three of 15 matches in all competitions.
Despite this, Carragher says Carrick can be proud of his tenure as he seeks new opportunities elsewhere after grabbing experience at one of the world's biggest clubs.
"Michael Carrick got the job on the back of nothing. He got the job on the back of basically playing for Man United and he's done really well," Carragher said on Stick to Football.
"So this idea that we've got to feel sorry for Michael Carrick. What's happened to him in the last 18 months, he will look back at this and say it's painful for him right now but what he did to get them into the Champions League, the big games he won against Arsenal, Liverpool, all these games, he's done brilliantly."
He later said: "But I'm not sat here feeling sorry for Michael Carrick. He's not a manager who's been managing for 10 years, working all over the world to get this job at Manchester United - he had about two years at Middlesbrough, was doing a bit of punditry, then got the Man United job. They gave him it.
"I don't feel sorry for Michael Carrick. He's been managing one of the biggest clubs in the world on the back of basically not really doing a lot. He's got them in the Champions League, he's done really well, but he was never at the calibre of managers in the Premier League - the managers in the Premier League are the best in the world and he's just not quite at that level."
Before the collapse this season, Carrick had a better win rate than any other United manager since Ferguson. He will now take the next step in his career, with many clubs likely interested including Crystal Palace who also sacked Pierre Sage today.