The 20-year old Tottenham defender spent last season on loan with the Swans from Tottenham and impressed so much in the back four that Neville feels he is a bigger loss than summer departures Joe Allen (to Liverpool) and Scott Sinclair (Manchester City).
"They've lost a couple of really good players - Sinclair and Joe Allen," Neville told reporters after the Toffees beat Swansea 3-0 on the weekend.
"Caulker - I know people talk about Joe Allen - but I think Caulker was a massive rock at the back for Swansea last year.
"I've had the pleasure of working with him for a short space of time [as coach of] the England Under-21 team. He really impressed me - (he's an) out and out defender. He was technically good on the ball, but he did like to defend.
"And I think out of all the players that Swansea have lost this season, maybe Caulker was the one that they probably miss more than anything because in the Premier League forwards are getting quicker, better technically and goal scoring this season has been frightening.
"And I think if you keep clean sheets you're going to win most games and at the moment Swansea are probably finding that difficult."