The npower League One team stunned their Barclays Premier League hosts by taking a 2-0 lead in the first half at the Britannia Stadium thanks to a brace from James Collins.
Stoke rallied after the break as goals from Kenwyne Jones and Jon Walters took the contest into extra-time, but Swindon would not be denied as Aden Flint put them 3-2 ahead before Collins responded to Peter Crouch's equaliser by completing his hat-trick with a minute remaining.
"It is obvious that when you win against a side like Stoke City away from home - in a place where teams like Arsenal and Manchester United find it difficult to score even one goal and you are playing in a fortress - it is something special," West Ham United legend Di Canio said.
"We scored four goals and I don't want to exaggerate, but I think we created another five or six clear chances to score.
"I don't think they underestimated us and all the congratulations I received from (Stoke manager) Tony Pulis at the end of the game was special for me because he is an experienced and very good manager who has achieved fantastic things with Stoke.
"So I am very proud. It is only one match, but to win 4-3 away from home against Stoke City is a special moment for my club, and I think one of the best performances in this club's history."
Collins joined Swindon this summer from Shrewsbury and Di Canio feels the 21-year-old striker could have a big future ahead of him.
"I wanted him desperately in my team and he is a fantastic talent," Di Canio said.
"He is a genuine and humble guy who always listens and asks for information because he wants to become a real footballer; not that he isn't one at the moment, but he really wants to arrive at the top and he has enormous potential.
"He is young, but he is a real warrior and he deserved those three goals."