The Blues' poor away form continued as they were second best at the City Ground, despite going ahead through Raheem Sterling's first Premier League goal since August to eventually draw 1-1 with Nottingham Forest on Sunday.
They were outplayed in the second half and saw Forest deservedly level through Serge Aurier as they made it four games without a win on the road in all competitions.
Both under Potter and his predecessor Thomas Tuchel they have struggled in tough environments and Potter knows that is something they have to learn to deal with.
"That's the Premier League for you," he said. "Away from home teams are not going to make it easy for you.
"On one hand you have players asking if they can have the players' shirts and before the game they are thinking they are going to run through a brick wall to prove they can be good enough to beat Chelsea.
"With that comes the responsibility on us. We have to be humble enough to know that is the challenge we face and we have to do better."
The visitors had the chance to take the game by the scruff of the neck after Sterling's early opener, but failed to capitalise and in the end were perhaps fortunate to leave with a point.
"Overall when the game was controlled for us in the first half I don't think we did well enough," Potter said. "Our level wasn't good enough, we didn't move fast enough, didn't have enough movement.
"You have to credit the opposition - they played a good version of themselves. I think it would have been unfair if we had taken more than a point.
"You look at their recent results here - they make teams suffer, and we suffered today for sure.
"Over the course of the game, understanding the responsibility we have, we could have done better."
Forest followed up the Premier League win over Liverpool and the Carabao Cup success against Tottenham with another positive showing against one of the big boys to raise hopes they might be able to beat the drop this season.
As well as Aurier's well deserved equaliser, they also hit the woodwork through Morgan Gibbs-White while Brennan Johnson spurned two good opportunities.
It left head coach Steve Cooper with a feeling of what might have been.
"If any team deserved to win, it was certainly us," he said. "If you look at the chances in the game, they were all ours.
"We were unlucky with the goal. It hit the bar and fell to Raheem a few yards out - it could have gone anywhere. That was unlucky because I never thought we were going to concede.
"Second half we didn't change but we were better timed with our press and we could get up the pitch. We were the superior team - no-one can take that away from us.
"Mixed feelings on not winning, but we have to be realistic on who we were playing and the team they are and take positives from the good things we did. I am sat here pleased with the performance but also thinking it was a game we should have won on the day."
Forest have suffered a blow with news that Jesse Lingard will be out for a month with the hamstring injury he suffered at Manchester United in midweek.
"Jesse is out for a month," Cooper added. "He has a hamstring injury, so it is as simple as that really."