Tottenham made it the club's worst ever start to a league season following the 2-1 defeat at Stoke. Spurs had two players dismissed while defender Vedran Corluka suffered an apparent serious head injury.
Boss Juande Ramos said: "I am worried about the situation but last season we won the Carling Cup and everyone was very happy.
"At this moment everyone is very sad and I am worried about it as I don't like this situation.
"We need to work very hard to change this situation and only hard work can change it.
"This is my job and I am working very hard each day to finish this situation.
"The responsibility is all the people working in the club starting with me and finishing with the last person - we all have responsibility, when we win and when we lose."
Asked if he planned to meet the chairman, he added: "I don't know - this is a question for the chairman.
"I speak to the chairman when he permits us to speak. He's a very busy person and I don't know when it will be possible to speak."
Ramos continued: "Sometimes football is difficult to explain. We had two penalties against us, two players sent off, some players injured.
"It was very difficult for us after the sending off in the first half but the attitude of the players was very good."
He also said there were positives to take from the last two defeats.
"Against Hull and today I liked the team but sometimes it's difficult to play with 10 against 11.
"I think the penalty decision was okay but the (Bale) red card should have been a yellow card."