Casillas left Real for Porto last summer.
"I had to decide whether I was going to stay and be the figure of ridicule that others enjoyed poking like a voodoo doll, or leave and be able to breathe again. I chose the second option to live more peacefully," he told XL Semanal.
"If I had looked out for myself more, it would have been a madhouse every day, particularly in the last year."
Asked about his feud with Jose Mourinho, he added: "It was uncomfortable because within the team there was two people whose relationship was deteriorating - it's something that happens all the time - it happens to a lot of couples. But that bad relationship made people take sides - it divided people. I only hope that, with time, those ghosts of the past will eventually disappear."