The Belgian winger was sent from the pitch in the 80th minute of the semi-final second leg, which ended 0-0, allowing the Swans to progress to the final where they will meet Bradford City.
The ballboy was attempting to pick up the ball after it had gone out for a Chelsea goal-kick but Hazard hoped to retrieve it quicker and in the end, forced the ball out from under the youngster with a swift boot, prompting referee Chris Foy to produce a red card.
Williams admits he seeing the contact being made: "Demba Ba told me the ballboy held on to it.
"I saw him kick him in the rib and you can't do that to a young boy."
Despite the sour end, it was a great night for Swansea who managed to make it through to their first major cup final in club history.