Potter has agreed to take over the reigning club world cup champions on a five-year contract, bringing his three-year spell with Brighton and Hove Albion to a close.
Chelsea ruthlessly sacked Thomas Tuchel on Wednesday after the club's underwhelming start to the season culminated in a 1-0 loss to Dinamo Zagreb in their UEFA Champions League group stage opener on Tuesday.
New owner Todd Boehly has settled on Potter to take the club forward.
Chelsea were noted for chopping and changing managers under previous boss Roman Abramovich but even under new ownership, old habits die hard.
"It's a very ruthless club and I'm not sure [Graham Potter] is the right fit," Carragher said on Sky Sports.
"Maybe Chelsea are saying they want to do things differently in the future in terms of having a coach in charge rather than a manager or they do things behind the scenes differently. But if I was Graham Potter, I would find that very hard to believe that they're going to do things differently. They've just got rid of a manager six games into the season.
"It's a huge step up in his managerial career and Chelsea are still one of the top clubs. They are still world champions right now, but you fear for Graham Potter that he'll be in the same boat as every other Chelsea manager has for the last 10 years - unless he gets results instantly they'll be getting themselves another new manager in 12 months."
