Ferguson was left disgruntled that Robin van Persie was not awarded a free-kick after being challenged by Chico Flores just moments before Swansea equalised, and the Scot was openly critical of Oliver's officiating.
"It was a foul on Van Persie, a clear foul," he said. "The referee had one of those days where we didn't get anything out of him.
"It was a shocking performance. I know he is a young referee but I was disappointed with that performance."
But Ferguson was not critical of his players, despite their failure to find a winner during a final half-hour where they put Swansea under incessant pressure.
He said: "We absolutely battered them. It was a great performance in the second half. It is unbelievable we didn't win the match.
"My only criticism is that the final ball at times let us down. "No criticism of the performance, it was very good."