The 19-year-old recently signed a new five-year contract at Stamford Bridge after returning from a loan spell at Dutch side Vitesse, where he scored 11 goals in 34 appearances across all competitions last season.
Broja has been at Chelsea since the age of nine and has come through the youth ranks to reach this far.
He made his senior debut for Chelsea in a 4-0 Premier League win over Everton in the 2019/2020 season but that remains his only competitive appearance for the Blues.
“I feel very happy. I was really excited to join Southampton. It’s obviously a very good Premier League club,” Broja told Southampton’s official website.
“They’ve got a good history and they play in the best league in the world, so it’s amazing for a young player to experience it, be in and around a Premier League team, and learn and develop from the older players, the staff and everyone here. It’s a very good opportunity.”
Southampton boss Ralph Hasenhuttl added on Broja's arrival: “Armando is a very talented player who we were impressed by last season.
“He is young, but he comes now with a full season of experience in senior football with a good team, and he brings a lot of qualities that can help us.
“He is a strong, quick player with a good instinct and this makes him a good fit for the way we like to play.
“I know how highly regarded he is by Chelsea, and I think this will be a good place for him to continue his own development as well.”
Broja made his debut for the Albania national team last year and has won six caps to date.
Southampton start their Premier League campaign at Everton on Saturday.
