The Die Mannschaft took the lead through Gundogan's first half penalty but the 2014 World Cup winners were undone by two second half goals from Japanese players plying their trade in the Bundesliga.
Freiburg midfielder Ritsu Doan equalised with 15 minutes to go before Bochum forward Takuma Asano found a winner for the underdogs seven minutes from time.
Gundogan was left incensed by the manner Germany conceded their goals and did not hold back in his criticism of Hansi Flick's side post-match.
"We made it too easy for Japan, especially with the second goal. I don't know if an easier goal has ever been scored at a World Cup," Gundogan told reporters.
"It can't happen. This is a World Cup. We dominated the game for the most part, Manu [Neuer] saved us once in the second half. That's why we have him.
"We had incredible opportunities up front, but we didn't score the second goal. How the goals came about was far too easy.
"There was a bit of a lack of conviction that we could hold the ball and move, that we could offer ourselves.
"We relied on long balls in the second half and the short balls we played were lost easily and quickly. You felt like not everyone wanted the ball."
Germany next face a Spain side, who smashed seven past Costa Rica in their opening game, on Sunday.
