Chelsea have now won just one of their last nine Premier League games. Potter also insisted Joao Felix's red card on debut was not malicious after he was sent off for a rash challenge on Kenny Tete.
The score was 1-1 when the 23-year-old was sent off, after Kalidou Koulibaly bundled the ball over the line to cancel out former Chelsea winger Willian's opener for the hosts.
But Carlos Vinicius' header secured the win for Fulham and only their second victory over Chelsea since 1979.
"It's really frustrating, it's incredibly challenging, it's a tough moment, I feel for the supporters," Potter said.
"We're disappointed to lose tonight I thought it was a relatively even game in the first half.
"We had a couple of good chances, but I think the basics we can do a little bit better in terms of our defending and our football actions
"At the same time we get back in the game and then the red card, it's where we're at at the moment where things can happen for us. Obviously losing Denis (Zakaria) beforehand as well, so we just adjusted and then Joao was obviously having a fantastic impact on the game and you could see his quality.
"So, to lose him, and then to lose him for three matches it's just really disappointing."
Felix now faces three matches on the sidelines, to add to the lengthy list of players unavailable to Potter.
The Portugal international almost had an instant impact, with some clever wide play to set up Kai Havertz, and the ball fell to Lewis Hall after a rebound, but his shot was straight at Bernd Leno.
However, Potter does not think their was any intent in the challenge that saw Felix dismissed.
"All of them will be learning moments. Joao is a young player. He's a top player, you can see his quality, you can see what he's brought to the game," the Chelsea boss said.
"I don't think there was any malice in the challenge at all, it was a little bit high of course and then the referee has got a decision to make…he (Felix) will learn from it.
"Like I said probably more than anything (it was) fantastic from the opponent it was more (of a case of) basic actions that can be done better and that's what we have to improve."
The result lifted Fulham up to sixth, four points off fourth-placed Manchester United, after a fourth consecutive victory, and Marco Silva believes it was a significant moment for the fans.
"Overall I think it was a great night for our fans and they deserve it. Sixteen years without a win in this game, and Chelsea superiority in these type of games has been huge," Silva said.
"We know they have completely different goals to ourselves, they are a massive club and their budget is completely different - we cannot compare anything but we knew that on the pitch with our work, with our strategy, with our identity, we could balance a bit more things and we did it.
"I'm sure that it was a great night for our fans and they deserve it."