United are at Anfield on Sunday.
Carragher told the Daily Mail: "First I was brought up as an Evertonian. The derby was a big part of my life and still is now.
"The rivalry with Man U is big because we are the two most successful clubs in England. The whole country is looking at this game. But, for me, I've never seen the derby as being against Man U. It's against Everton. When you get beaten by Man U, you don't see their fans.
"When you get beaten by Everton, their fans are all around you. That's the difference. You can go into town the day after you lose to Man U but if you lose to Everton you shouldn't be seen anywhere if you are a Liverpool player."