Yet Big Sam reckons rookie ref Anthony Taylor caused the problems by failing to send off goalkeeper Schwarzer early on.
Boss Allardyce said: "A lot of Southern-based columnists never even watch Blackburn play so what they say doesn't really bother me.
"Has nobody ever fouled a goalkeeper before? Yes it was a foul and the referee should've blown his whistle.
"There were four major decisions on Saturday and three went against us, maybe one went for us."
Big Sam reckons most of the problems in Blackburn's draw with Fulham stemmed from Taylor's failure to send off Schwarzer when he stumbled outside his area and got a glove on Morten Pedersen's chip.
Big Sam said: "The blame lies, not with El-Hadji Diouf, but with the referee. He missed a blatant handball from Mark Schwarzer after seven minutes.
"That would have cured all the problems that happened throughout the game."