Karanka joins a coaching team consisting of Rui Faria, Silvino Louro and Jose Morais and, like the trio, has also signed a four-year contract with the Spanish giants.
It is thought his appointment came about because Mourinho wanted someone who - like Giuseppe Baresi at Inter Milan - knew the club well.
Karanka spent five years at Madrid, between 1997 and 2002, helping them to three Champions League trophies and one league title.
The 36-year-old was serving as Spain Under-16 coach, while he was also assistant to Gines Melendez with the Under-17s who have just returned as runners-up from the European championships.
Madrid have thanked the Spanish federation for the ease in which they allowed the Primera Division club to sign the Basque coach.