Potter's top-flight tenure with Blues got off to a flying start with a dramatic 2-1 victory over Crystal Palace to begin October, the first of three consecutive Premier League wins for the former Brighton boss.
But their domestic form has been a mixed bag since, only managing to muster draws with Brentford and Manchester United before a disappointing 4-1 loss to his former club last time out.
Potter said: "If you look at Pep [Guardiola]'s first year, if you look at how long it took Jurgen [Klopp], I mean these guys are top, top people, but even they don't just walk in and click the fingers and everything's perfect.
"Sometimes we don't do well enough, so [criticism] is justified. It's a combination of lots of things. It's part of the circus and the noise that makes the Premier League so interesting.
"As a coach, you have to understand the short term, medium term, long term. If you just focus on the medium and long, you don't get the short term and you're in trouble."
Potter has in many ways been forced to operate primarily in the short term. While his five-year contract at Stamford Bridge was announced in September, he had to contend with an international break, then what were set to be his first two Premier League games with Blues were postponed due to the Queen's death and funeral.
He will now face another extended spell apart from his squad with just two Premier League matches and Wednesday's Carabao Cup clash with Manchester City remaining before the World Cup.
Sunday's meeting with high-flying Arsenal could be Potter's toughest test yet as his seventh-placed Blues look to improve their standing before the break.
Arsenal, meanwhile, could reclaim their place at the top of the table with a win to extend what has been one of their best starts to a Premier League season.
Asked if opposite number Mikel Arteta - who signed a new three-year contract with the Gunners in May - might serve as inspiration, Potter replied: "This is just the beauty of life. People think that it's just going to happen for you.
"This is incredible, at what level does that happen? But I think it's quite a dangerous message to tell people, but it is what it is," joking, "Alex Ferguson took his time, I think, if I remember right, and he did quite well.
"Pep didn't just walk in and Pep's a genius. Jurgen took some time. The evidence is there but obviously if you want to use another narrative that's fine. I can't control that.
"The team that I left at Brighton wasn't the team that I started with at Brighton. It grew and it developed. Sometimes you fall on your face and you have to pick yourself up. You have to just deal with that. That's the life of a coach."