Wagner has been linked with the Blues post in the event Antonio Conte leaves, but the German insists he won't talk to any club behind Huddersfield's back.
Wagner told Kicker: "Chelsea and Frankfurt interest? I take it for granted.
“I speak over every decision with the owners of our team.
“We have experienced a lot of memorable moments together, we respect and support each other.
"Our aim is to stay in the Premier League. I am not interested in what the press is talking about."
Wagner's Huddersfield sit 16th in the table with four games to go.
Crucially, though, they have a six-point lead over 18th-placed Southampton.