Altidore came closest to an equaliser in the 65th minute when he was picked out by John O'Shea in the box and swivelled past Aaron Hughes before firing narrowly wide of target.
Altidore admitted on Sunderland's official website: "We just have to do better mentally at not giving away the game. It's difficult but also we have to create even more chances.
"This is the best league in the world and that's why it can be so cruel but at the same time we have to take the positives away and look forward to the next match and try to get some points."
For the big stories and the banter follow us on Twitter: @tribalfootball