Potter endured a chastening afternoon at the AMEX Stadium on Saturday as Chelsea were battered 4-1 by Brighton.
The defeat was Potter's first in 10 games as Chelsea manager since leaving Brighton for Stamford Bridge in early September.
The 47-year-old was in charge at Brighton and Hove Albion for three seasons, leading them to their highest place finish in the top flight(9th) last season.
"I have nothing to say sorry for, I've nothing to apologise for," Potter told reporters when asked about the reception he got.
"I did a good job, you can see the team's a good team. I took over when they were fourth from bottom in the Premier League, probably third-worst team [in the division].
"There's a lot of money been raised in terms of player sales and there's a lot of good players on the pitch.
"I hope for their sake the next manager does as good a job and that's as brilliant for them."
Two own goals from Ruben Loftus-Cheek and Trevor Chalobah did not help Chelsea's course, with Leandro Trossard and Pascal Gross also scoring for the hosts and Kai Havertz getting a consolation for the Blues.
"Of course, the scoreline and the defeat is a painful one," he continued.
"We didn't really recover. Once the crowd were so engaged it was a difficult one for us. The scoreline was painful. Brighton played a good game, especially in the first half.
"You can always look at goals and think you can do better. We have to take responsibility a bit in the first half.
"We have had a lot of football but that's no excuse. We couldn't quite get to that level. We didn't take our opportunities in attack and they did.
"We had some opportunities to attack them too but unfortunately we missed the pass or didn't execute, then the scoreline was a painful one. When you lose you can always do better."
