The Timbers hit the lead through Kris Boyd in the 16th minute before Alejandro Moreno and Nick LaBrocca put the visiting Goats ahead for an eventual victory on the road.
Spencer was disappointed with his players after the match, suggesting that they have to stand up and be counted.
"A few guys in there need to look at themselves tonight, and ask themselves, are they doing enough on and off the field?'' he said angrily.
The loss was Portland's third in a row, pushing them down to second bottom of the Western Conference table, one point ahead of reigning champions LA Galaxy.