Despite finding success in the MLS with several teams, and in Holland, Altidore did not enjoy his time in England.
He struggled to score goals and could not justify the transfer fee the Black Cats paid for him.
"It didn't feel right. The timing was not right," Altidore told VICE.
"I remember very well that Gertjan Verbeek and Earnest Stewart said that I had better wait for other offers, because Sunderland did not play my type of football. They were very honest about that, saying it would be a disaster if I went to Sunderland despite the transfer fee being so big.
"But my father really wanted me to go to England because the contract was good there. What if I suffered an injury at AZ? Then I might not have been able to get that contract anymore. So it was a difficult, emotional decision. I went to Sunderland."
He added: "My agents and my father thought about finances. With the contract at Sunderland, I was able to secure my future, guaranteed.
"I was a young man. If I had had the knowledge that I have today, I would have made a different choice."