Swansea agreed to Boro's offer of money plus Marvin Emnes, but Graham did not want to drop out of the top-flight and opted for Sunderland.
"Boro was an option, but no more than that," said Graham.
"Of course, Middlesbrough was my first club and so there was still a pull there, but I really wanted to carry on playing in the Premier League.
"The chance to return home to the North East was a massive attraction for me.
"I have been away for a long time, living and playing in Watford and in Swansea.
"So to come home and play for a big football club up here is great and once Sunderland came in for me and I had the chance to play in the Premier League at a club of this size, it was always going to be this club that I wanted to move to."