James said: "You pay your way and you are entitled to your opinion."
The Pompey keeper reckons fans have a right to dish out stick as long as it stays within the law.
Striker Defoe and boss Redknapp quit Pompey this season for Spurs and police and both clubs fear tension from the terraces when the sides clash at White Hart Lane on Sunday.
But James said: "As long as it is not racist or homophobic - give stick.
"As Isaac Newton said there is an equal and opposite reaction to every action."
Pompey insist: "Any fans involved in indecent chanting will be dealt with - with the possibility of a lifetime ban."