He said: "Having started my career at Arsenal as a youth player, and then as a coach, I know the history, class and values of this great football club.
"I can't wait to get started in my new role and to meet the players and my new colleagues.
"I would like to take this opportunity to thank Dave Jones and everyone at Sheffield Wednesday for their support and understanding."
Gunners boss Arsene Wenger said: "I have known Terry for many years and he is someone who is hugely respected and highly regarded in the game.
"Terry brings with him a wealth of coaching experience and he will be a great asset for us, working with our reserves and youth development players."