Brown worked with Altidore at Hull City.
He told the Daily Star Sunday: "We took over a loan he was on from Villarreal in Spain. He was only 19 - a pup - but he had potential.
"He has started to fulfil that potential. He did well in a good league in Holland. For a big lad, he wasn't as good in the air as you would have expected. It was something he had to work on.
"But he was gifted. He was a beast in terms of his physique. He had pace and good technical ability.
"I would say £6m is not a bad price for a 23-year-old with Altidore's attributes.
"If I was signing him now for the Premier League, I'd be looking for him to get me double figures in terms of goals this season - then 20 in his second year."