The Senegalese striker clashed with Brown after the Hoops skipper scored the equaliser in the 2-2 draw in the fifth round of the Scottish Cup at Ibrox on Sunday.
After the match, Diouf said that Brown tried to get him sent off but the 30-year old, on loan from Blackburn Rovers, claims he was the bigger man in the situation.
"I refused to react. Why should I react to him? No one knows him - I'm too big and too strong for him," Diouf said about the incident in which Brown squared up to him after he scored.
"Brown tried all sorts of things to get me sent off.
"He said some nasty things about me and my family on the pitch - but I am too big for him.
"He can say all the nasty things he wants. I won't react.
"He just wants publicity because if I react then people are going to say that El-Hadji did something wrong."