Spence was speaking from England's training camp at St George's Park after a first Three Lions call-up.
Conte led Napoli to the Serie A title last season, though had a difficult time when in charge of Spurs, which saw him fall out with Spence.
The player recalled: "I would probably say the biggest hurdle is when I signed for Tottenham and I didn't necessarily feel the love straight away, with the manager and everything. I was on such a high going into the club.
"When I went there, I was on a high, and then I almost came crashing down because of comments from the manager and not playing. I was in a difficult space then.
"I am buzzing, I am going to sign for Tottenham. So to hear comments like that wasn't nice. It does shatter your confidence a bit. But, you know, I am a fighter. So, whatever I do I will try to do my best in everything I can. Fast forward, I am here now, so I am happy.
"I wouldn't say I particularly have a list or anything, but I do have a mental note of people who doubted me, for sure, and it does feel good to prove them wrong. Obviously, at the time it wasn't nice but I feel like I don't put my energy into certain things like that. I just focus, do what I need to do. I've got a good family behind me that keeps me focused."