The Cottagers are expected to name a strong line-up to face the Polish champions as they look to avoid a repeat of their 1-0 defeat in Poland.
"You never want to lose and because the table is so tight we need to win," Dempsey said.
"We need to get points as you don't want to be waiting until the last game to get our of the group stage.
"We're going to do everything we can to win the game. While the league is most important, so too is this competition as you want to be a part of as many competitions as you can.
"To play in these games gives you great memories.
"I mean getting to the Europa League final the last time we were in it is something I will never forget and hopefully we can have a nice little run.
"It is nice to be able to play in this competition as it allows you to play different teams from all over [Europe]."