Fellow new arrival Cesc Fabregas also featured in Jose Mourinho's side for the first time in Sunday's clash at the Stozice Stadium, and it was his through ball which allowed Costa to cancel out Nik Kapun's 40th-minute opener with an emphatic finish.
The 25-year-old, who penned a five-year deal from Atletico Madrid this summer, raced onto Fabregas' pin-point pass and fired into the roof of the net in the 55th minute.
Kurt Zouma then added a second from close range on 63 minutes, after Fernando Torres' strike was parried into his path for a simple tap-in.
Branislav Ivanovic thought he had made it three, only for his effort to eventually be ruled out, while Torres himself missed a glaring opportunity when he diverted Nathan Ake's left-wing cross wide from inside the six-yard box.
For the big stories and the banter follow us on Twitter: @tribalfootball