The number at grassroots level is at an all-time low - with an estimated 7,000 quitting every season.
And Ian Blanchard, the FA's head of refereeing, believes it is because more and more pub players are copying the antics of Premier League stars. He said: "A huge number are giving up because of the behaviour towards them.
"You can't blame it all on the professional game, but players' behaviour at the top level doesn't help."
"A headteacher telephoned me at Soho Square to tell me the behaviour of his pupils on a Monday morning often depended on what they had seen footballers doing on the television at the weekend.
"They knew their role models weren't behaving as they should and were copying that. Players have to realise that."
And Blanchard, speaking at the launch of the FA's Get Into Refereeing scheme which aims to recruit 8,000 new refs by 2012, added: "That incident didn't do us any favours.
"We've got some fantastic players in the Premier League - it's the best and the most-watched league in the world. The pressures of operating in the Premier League must be phenomenal for a player. They know they have to win.
"And there are hundreds of thousands of people who would love to be in Drogba's position. Players are under immense pressure to win the game. But we would ask them to please understand what we're trying to do.
"We've got a split second to make a decision, to decide whether the foul is inside or outside the area or to send someone off or not, so players need to give a bit of respect to the referee who is trying to do their best in what is sometimes a very difficult situation."