Brown joined the Millers on a season-long loan in August and made 20 appearances for the Championship's bottom club.
He joins Blues team-mate Kasey Palmer at the Terriers.
The England Under-20 international could make his debut for Town against Port Vale in the FA Cup on Saturday.
"He's one of the most exciting young players in the Championship in my opinion," boss David Wagner told BBC Radio Leeds.
"He has a lot of space to improve and he will suit our style. He is a perfect signing."