The veteran midfielder set up the only goal of the game, netted by Javier 'Chicharito' Hernandez, but was disappointed the Reds couldn't build on it in the next 80 minutes.
"Sometimes it can happen that way," said Giggs. "We scored an early goal and then we sat back. We shouldn't have done that really."
The Welshman acknowledged this trip to his homeland was a difficult fixture, with Swansea defying their newly-promoted status to pick up 13 points and concede only one goal at home - an own goal - prior to United's visit.
"It's a tough place to come. Their record this season shows that, they were unbeaten here until today. So we knew it would be tough."
Giggs accepted international breaks can impact negatively on clubs when their players return, but wouldn't cite this as a reason why United's 1-0 win was worryingly slender.
"You have different players at different fitness levels," he said.
"Some of our players hadn't had a game for two or three weeks and some, like Phil Jones and Nani, had two tough games.
"But that's not an excuse. We probably need to play better if we're going to get more positive results."