Cleverley won the league with United, but failed to fully establish himself at Old Trafford.
He told manutd.com: "When I look back on 2013/14, two games were massive in, well, my downfall, I suppose.
"Liverpool away at the start of the season, we got beat 1-0, and it was a flat performance from myself. I'd already had a man-marking job on Hazard in our previous game against Chelsea, and that day I had the same job on Coutinho at Anfield. For me to show the other side of my game, it was tough.
"Then you start hearing whispers.
"It's hard to get away from it all when you're living in Manchester, your mates are all United fans and you're playing for the biggest club in the world. Looking back, I probably wasn't mentally ready for everything that was going to come my way.
"I struggled with the mental side of things. In the mix of it all I had a couple of good games – the manager still had the faith in me to play and I played a lot that year – but then the Olympiakos away game killed me, really. I was poor that night.
"The snowball had already started, but now it had real momentum.
"I remember my next start, four weeks after Olympiakos, going into the City game at home when we got beat 3-0 and I've never, ever been lower on confidence.
"For me, confidence is the biggest attribute a footballer can have, and that night I had none of it."