France Under-21 international captain Kaboul made the mistake which gifted West Ham a goal in the 1-1 draw at Upton Park and then gifted Birmingham a soft penalty in last Sunday's 3-2 defeat at White Hart Lane.
"Younes made a big mistake against Birmingham and it makes it difficult for us - we need not to make silly mistakes like this," Cerny told setantasports.com.
"It was a silly foul and a penalty. And in the last few games we have made big mistakes. Younes is having a difficult time at the moment. But when he arrived he was playing well.
"I hope he has the character to come back and be a success at Spurs, because we have some very good players here.
"I think there is a problem with the confidence, not just of Younes, but of the whole team. When we are a goal up we need to relax and control the game more.
"How many times have you seen us concede a goal in the last minute? It's these little things that all add up. We have lost maybe 18 points in the last five minutes this year - that's a lot of points. I just hope our luck will change."