The Welshman is being linked with a £12million summer move to Liverpool to join former Swans boss Brendan Rodgers at Anfield but Britton is hoping he remains at the Liberty Stadium.
"We would be disappointed if Joe goes - I think that goes for the players, the staff and the fans," Britton told the South Wales Evening Post.
"That's because if he goes, we know we will be losing a really good player.
"Players like Joe are hard to replace - the money they are talking about him going for tells you how good he is.
"But you can't begrudge him the move if it happens. We would have to wish him good luck. We hope he will stay with us, but it would be a great opportunity for him if he ends up going."