The 53-year-old resigned last week to take over at Tottenham Hotspur and was criticised by sections of the Spanish club's fans for leaving for the money.
Ramos told Radio Marca that the €7m-a-year contract offered by the Premier League club was not his main motivation for quitting after guiding Sevilla to five trophies in the past two seasons.
"The situation became unsustainable because there are times when a club can trample on its coaches," he explained.
Several clubs were interested in his services during the summer, but he previously stated that he had turned them down because he did not fancy the move.
Now Ramos has revealed that Sevilla kept him from going before the start of the current campaign.
"Sevilla prevented me, not for any other reason," he added.
Despite walking out of the post at the Sanchez Pizjuan, Ramos admits that it was a very difficult decision to make.
"I left with some sadness after all the years I have been there," he stated.
Rumours that his relationship with president Jose Maria del Nido had deteriorated are wide of the mark according to the new Spurs boss.
He concluded: "It was always very good throughout all those years."