Albion had hoped to keep Mulumbu for the whole season after he retired from international football for the second time last summer.
But he has since resolved his differences with Congolese football officials and now says the chance to play in the African Nations Cup in South Africa is something he cannot turn down.
"I'm sad to miss West Brom games because it's my team," he said. "I hope when I come back we will still be in the same part of the table.
"I think it's a great season for us and we can do something really special.
"It's my country and I cannot let them down, I need to play for them and hopefully get through the group and maybe to the final.
"I'm looking forward to going to the African Nations Cup but I'm looking forward to coming back and being part of West Brom's season."