Crouch was dismissed in Wednesday night's Carling Cup quarter-final defeat for a lunge at Chelsea's Nigerian midfielder Mikel John Obi.
They called on the authorities to deal with the matter as a racist issue. But an FA spokesman said: "We would certainly not see Peter Crouch's comments as being racist.
The FA believe they cannot bring Crouch to book for expressing an opinion - and it was not a personal attack on Mikel.
SEM do not intend to let the matter rest and told The Sun: "SEM feel the comments were potentially racist to all foreign players, not just black or African ones.
"We are intending to speak to Trevor Phillips at the Equality and Human Rights Commission on Monday.
"We hope football's governing bodies, who are meant to be committed to stamping out racism issues, will also get involved."