Following on from Thursday's 1-0 Premier League win in the capital which left Chelsea in 10th place, City dumped Potter's side out of the FA Cup with a resounding 4-0 win at the Etihad Stadium on Sunday.
Riyad Mahrez opened the scoring with a fine free-kick before Julian Alvarez's penalty and a Phil Foden strike made it 3-0 before the break, and Mahrez finished the job with another spot-kick in the second half.
The half-time whistle was greeted by boos from the travelling support, and by the second half Chelsea fans were singing the name of former owner Roman Abramovich and Potter's predecessor Thomas Tuchel as they recalled happier times.
But Guardiola, who praised Potter when he was at Brighton, urged Chelsea to stick with their man.
"I would say to Todd Boehly, give him time," Guardiola said. "I know in big clubs, results are important but I'd say give him time.
"The second half is what he is. What he's done at Brighton is outstanding, but…we need time in the first season. I had (Lionel) Messi in Barcelona my first season so I didn't need two seasons because Messi was there."
Chelsea did not manage a shot in a first half Potter described as "painful", failing to even vaguely threaten Stefan Ortega's goal until a Mason Mount shot in the 54th minute which was deflected wide
He could point to an ongoing injury crisis which kept nine players out of this fixture and led to teenager Bashir Humphreys making his senior debut in defence, but Potter accepted Chelsea fans needed to see more.
"The results in a small space of time are not positive," he said. "You can make excuses and look for reasons or say it isn't good enough. Both of those answers are correct.
"We have to keep improving and stick together because clearly we are suffering as a football club and it's not nice at all. But that's where we are at the moment.
"We can't do anything apart from do our jobs better and work harder. You understand the supporters' frustration, we respect that. But our job is to do our job and keep working."
At least Chelsea are not due to face City again until they return here in the league in May.
"This opponent is probably the worst opponent you can play when things aren't going very well because they can make you look like you're not running or not trying because their positional structure is brilliant," Potter added.
"They use the full width of the pitch, they keep the ball, they make it difficult for you to put any pressure on them. I am working with the players, I don't get any sense of people with a poor attitude.
"Everyone wants to try and do better. I think there is support in the dressing room. We are just going through a bad moment and sometimes when you have these moments you need somebody or something to blame but we have to stick together and keep working."
Mahrez struggled for form early this season but has shown signs he is almost back to his best - and has certainly thrived against Chelsea this term, scoring four goals in three games.
"The quality from Riyad is everything," Guardiola said. "He loves to play football. He helped us to break the game with a fantastic free-kick. He played an outstanding game…
"Riyad in important games is a fantastic player, I know how important he is…Since we came back, he is playing really, really well."