Williams, 22, spent much of the past year on loan at Wycombe Wanderers, where he helped the club gain promotion from League Two last season.
England Under-19 left-back Brown moved to the Terriers on New Year's Day from Tottenham Hotspur.
Exeter City boss Matt Taylor hopes Brown's versatility can allow him to be more flexible with his line-ups.
"He is a dynamic left-sided defender. We have got the experience of Craig Woodman and Dean Moxey who can cover that position, but this lad is going to get up and down that touchline time and time again," he said.
"It gives me an option if I wanted to play five at the back because he can play wing back."