The Scots, with three points, still technically have a chance of progressing to the knockout stages at a World Cup for the first time as one of eight third-placed teams with the best records over the 12 groups.
As things stand, Scotland have the sixth-best record but Clarke clearly believes that they will be overtaken by at least two other teams with 20 group matches remaining.
"Fair result," he said.
"You give a team like Brazil the chances that we gave them in the game, you expect to get punished. And that's what happened. I think we're probably going home.
"We know they're deadly in the attacking third of the pitch, and we gave them for sure the first two goals, probably the third goal as well.
"So we gave them the goals, but on the counter of that, they also missed a few chances and Angus (Gunn) had to make a few good saves.
"We created one or two chances, but nothing really clean."
Clarke's frustration with his team's performance on Wednesday was clear but he said he was not angry with the players who got Scotland to their first World Cup since France in 1998.
"I'm disappointed for them because they didn't reach the levels that they can reach," he said.
"I think we all know that. Anybody that's watched this team over the last few years knows that we didn't reach the levels that we can reach."
The former Chelsea fullback said their campaign had also exposed structural issues in Scottish football that needed addressing.
"I think when you see the physicality, the power and the technique of both Morocco and Brazil, you can see that we have to do something about it," he said.
"We have to try and be better at producing young players that can grace the world stage."
Scotland's Tartan Army of fans have lit up the tournament and while Clarke once again saluted their contribution, he said it was important supporters did not forget why they had been in the U.S. in the first place.
"Don't forget that this group of players brought these fans to America. This group of players qualified," he added.
"Otherwise, we're doing what we always do. We're sitting on the sofa and we're watching the World Cup without Scotland.
"They've had a great time, great ambassadors for the country, but they wouldn't be here without that group of players."