The midfielder was widely tipped to link up with the new Kop boss Brendan Rodgers in the summer after starring for him on loan at Swansea last season.
The Icelandic ace said: “It was one of the hardest phone calls to make.
“Brendan knew me at Reading and knew me at Swansea and the things he did at Swansea in particular were fantastic.
“He is a great coach and I owe him a lot for giving me a chance at Reading and then giving me my first chance in the Premier League at Swansea.
“It was really hard to tell him I didn’t want to play for him this time.
“But I felt there was a lot going on at Tottenham with the new training ground and plans for another stadium.
“And Tottenham also finished fourth last season and were unlucky not to be in the Champions League.
“Ask any player and that is where he wants to be. Hopefully we will get there.”