Herd was sent-off for an alleged stamp on him in the 34th minute, which was spotted by World Cup Final assistant referee Darren Cann.
"Herd stepped back and then hit me on my thigh," Olsson told the WBA website.
"If you ask him, he will think it is a bit harsh, but I think he will agree it was a foul. It was a needless thing for him to do. When you do that in those areas, you get punished more.
"It is in the rule book - but how do you apply it?
"Last week against Wolves, the player was obviously grabbing me but the ref didn't think it was enough, not too bad, to give a penalty.
"They are not black and white those things. It is always a question of how you apply the rules.
"The linesman saw it, he was in a good position, and decided to act on it, and credit to him for taking a decision like that.
"I said in an interview after the Wolves game the level the refs are keeping is quite consistent and sometimes you have decisions against you and sometimes they go with you.
"I don't think it changes the game too much because our two goals were from set pieces and I don't think you are too bothered defending set pieces with one guy down."