Just 12,700 attended the 2-0 defeat for the Baggies.
Pulis said: "There's got to be some way of filling stadiums up, especially for third round and fourth round ties at Premier League grounds.
"That has to be looked at because it is a wonderful competition.
"For all of what is going on, we've go to try as a nation to make sure we keep it a great competition by filling stadiums for games - that's certainly a help.
Pulis added: "It's difficult because it's just after new year, people have spent a lot of money, so you understand that.
"But, as I said, there's got to be a way at this time of year that clubs can attract full houses, how they do it you'd have to ask chief executives."
For breaking stories and all the great banter like us on Facebook: facebook.com/tribalfootball