The Blues are reportedly set to sign the French midfielder in a £35m move in the coming days.
Bakayoko's arrival at the West London would pave the way for Nemanja Matic to join Manchester United.
However, Smith is certain that Chalobah would find very limited playing opportunities in Antonio Conte's side next season.
"You're thinking he will probably have to leave. That goes right across the board with a lot of the boys.
"People like Alfie Mawson, no disrespect, but at a slightly lower level of team in Swansea can have a career."Jordan Pickford, obviously going to Everton, that's a brilliant move for him. It is really difficult, Germany are the same."