The England winger signed for the Hammers last month from Wolves.
Jarvis told The Sun: "I was at Millwall for about six years and then released before getting a youth-team contract.
"Being told I was too small to make it did not hurt nearly as much as being told simply that I wasn't good enough.
"But I have used it to my advantage, you thrive on trying to prove a point that people were wrong.
"I was Surrey county champion at breaststroke for four years from 12 to 16. Football was always my main sport though, right from the age of four.
"Luckily, after being released by Millwall I was given a chance by Gillingham. Even though my trial was a disaster I got taken on and after a year as a youth-team kid there I was in with the first-team squad."