So far in their debut season, the Timbers have conceded goals basically as soon as they score which has restricted them to just two wins from their six matches.
"We don't want to get (tagged) with a stigma that we're (mentally) soft," he said.
"It does eat at me, because I take this very seriously. As players), you've got to look at yourself individually and you've got to analyze yourself.
"I know if I was playing against (the Timbers), that's what I would say as a coach, 'If you get one (goal) against this team, there's a chance you could get two or three,' " Spencer added.
"You don't want to get a reputation .we need to try and erase that as quick as we possibly can."
Spencer will be hoping his Timbers can get this part out of their game for the remainder of the season starting with Real Salt Lake on the weekend.