The Scotland midfielder will now serve a three-match ban for his challenge on Fernando Torres.
Baggies boss Roberto Di Matteo said: "We've looked at all the video replays.
"Yes, James caught their player but he made a genuine attempt to win the ball.
"We feel the punishment outweighs the crime.
"That said, FA rules require us to prove the ref made a 'clear and obvious mistake' and we consider it unlikely a panel would agree in this instance."