The England international admits Saints chiefs were concerned about his height.
He explained to the Mirror: "People at school used to assume I was going to be a footballer and it wasn't until I got to 16, when I was at Southampton, that I had a doubt for the first time in my life.
"Southampton said I wasn't big enough but it was just because I hadn't grown. Simple as that. Some boys grow quicker than others. From the age of nine to 12, I was one of the higher achievers in the age group.
"But everyone shot up at 14 but I didn't shoot up until 16. So during those two years, when I was coming up to 16, Southampton said: 'We don't think you can cope'.
"They were going to release me, but then said: 'Technically you are one of the best players in the squad - you just can't affect the game'. That's what they kept telling me.
"Southampton told me I had two months to get a scholarship. They told me there was a lad who was going to get it ahead of me. That was the first time I realised I might have to say to my friends: 'I am coming to college with you lot'.
"I was the last player to get the scholarship in my age group. They gave me two months over Christmas, so I went away. I was doing gym and extra running, and doing loads of stuff with my dad.
"At the same time, I was going to look at colleges, thinking: 'Oh I don't want to go to college, I want to be a footballer'.
"But I came back, got my scholarship and, six months later, was in the first team. When it levels out and you get to 17, 18, ability takes over again to a certain extent and that is when I pushed on."