Hakan Calhanoglu struck the only goal of the game on the stroke of half time as Barcelona dropped to third position on the table.
Barcelona had a goal from Pedri disallowed for handball by Ansu Fati in the build up following a VAR review.
Inter Milan full back Denzel Dumfries was left off the hook later on in the second half when the ball appeared to strike his hand from a cross but following another VAR review play was allowed to continue, much to the anger of the Barcelona manager.
Barcelona are now in danger of failing to qualify from their Champions League group for the second straight season ahead of facing Inter Milan again next week at the Camp Nou.
Xavi did not hold back in his criticism of referee Slavko Vincic and his VAR team after the game.
"I am outraged," Xavi told reporters.
"First they explain to us that Ansu Fati handled but another team-mate scored, then with the other incident, it is not clear what happened.
"It is my opinion, I would have liked to speak to the referee, because he did not blow the whistle. At the moment, I am outraged, it is an injustice and it makes no sense.
"Now we still have three finals left, we have already lost in Munich and we start again. But there is indignation.
"In general, it was a great injustice. I can't hide and say I'm not outraged, it's a great injustice. The referee should give explanations, instead he goes away and nothing happens. He has to come here and explain."
Xavi has become the first Barcelona manager to lose his first three Champions League away games.
