Darijo Srna broke the deadlock on 24 minutes when he beat Marcell Jansen to Danijel Pranjic's cross and fired home.
Croatia were gifted a second goal on 63 minutes when Ivan Rakitic's cross came off Lukas Podolski and rebounded off the post for Ivica Olic to slot home.
Podolski pulled a goal back with a late drive but Bastian Schweinsteiger's red card ended German hopes in injury time.
It was a famous victory for Croatia, and an echo of their 3-0 success over the same opposition in the 1998 World Cup quarter-finals, but it casts serious doubts over Germany's credentials for the European crown.
"It was a great game. But I want to congratulate my players - not only for the fight they showed but for playing an excellent match," said Croatia coach Slaven Bilic.
"There is no other way to beat the Germans than to play well.
"We controlled the game but Germany are still a great team and still one of the favourites for the title."