Traore, along with Pompey team-mate Sean Davis, has been linked with a move to the Premier League new boys.
He spent the second half of last season on loan at Rennes and is expected to leave Fratton Park on a permanent basis over the summer.
"I am relishing the challenge, possibly bringing eight to 10 new players in," said Brown.
"I spoke to Djimi and I was interested, but Djimi made reference to wanting to stay in France.
"He would, of course, have come to England if we were available as a second choice, but I don't want that kind of attitude. I want first choice."