The Hammers were beaten 3-0 at home by Southampton on Sunday which was their fourth loss on the trot and fifth in six Premier League matches this season.
The negative result further compounds recent 4-2 defeats to both Watford and West Bromwich Albion which prompted a strong response from the boss on a day that was again marred by crowd violence at the London Stadium.
"That's not a good sign. It hurts me as a manager and it hurts the team. It's my responsibility, it's my team, I know how it works and I'm not running away," he told Sky Sports.
"These are the same players, and the same manager, as last year and at the moment everything that was really good last year is bad this season and everything that was average last season is now way below par.
"The season has only started, it's true, but it's happened for four games now and we have to change it big time.
"I told the players after we lost to West Brom that there is no need to panic, the credit was there for a lot of them. But now that credit is gone."