Brown played a key role last season in helping Town earn promotion into the Premier League and ahead of the Seagulls' clash against the Terriers this weekend, the youngster has revealed that he is looking forward to facing his former team.
Brown revealed: "After the play-off final I thought I was going to go back. But, as the summer went on, things didn't start to work out the way it was meant to be.
"They wanted to buy me on a permanent and it wasn't possible. I had a few meetings with the manager here and thought this was the club I wanted to join.
"The way the manager plays will improve me. He's a great coach, so I made my mind up early in the summer that Brighton was a club I wanted to go to.
"All the (Huddersfield) fans are still tweeting me, saying come back, come back, but my main focus is on Brighton. So when it comes to the weekend I want us to win. That's all I want to do, it's a must-win game for us."