Even though the British pair led for more than half the match, Norwegian duo Kristin Skaslien and Marcus Nedregotten managed to hang on for a 6-5 win after scoring three points on their power-play in the sixth.
Dodds and Mouat will now have to prepare for a bronze medal match against Sweden on Tuesday afternoon.
Mouat and Dodds did not hide their disappointment in the wake of the contest but immediately sought to shift focus onto Tuesday afternoon's match against Sweden, which will decide whether they make it onto the Olympic podium.
"We're disappointed not to be in the final but there's still a bronze medal to play for tomorrow," said Dodds. "I think if we play like we did for the majority of that game it will be a really good fight against Sweden."
Dodds and Mouat had lost their penultimate round-robin match to the Norwegians 24 hours earlier, but rebounded to secure a swift chance at revenge when they overcame the already-eliminated Americans on Monday morning.
Having trailed early in their round-robin tie, Britain made a much stronger start, taking the first end against hammer - or final stone - after Skaslien missed an attempted take-out.
A clever hit-and-roll by Dodds with the last stone of the third gave her a 3-1 lead and Norway were forced to settle for a single in the next, giving the Britons a 3-2 lead at the half-way stage.
Dodds had a good chance to score heavily in the fifth, but narrowly mis-judged her attempt and deflected off a Norwegian guard, forcing her to take one and relinquish the hammer going into the sixth.
Dodds was made to pay for an inaccurate take-out attempt in the next, giving Skaslien the chance to take full advantage of the Norwegian powerplay and roll in for a three and a 5-4 lead.
Britain could only respond with a single point from their own power-play in the penultimate end, hauling the tie back level at 5-5 but giving the Norwegians the hammer advantage in the decider.
And despite Britain's best efforts the match came down to Skaslien's final shot, the Norwegian calmly rolling her final stone into the centre of the house to signal heartbreak for the British pair.
Coach David Murdoch called the duo's half way lead their strongest passage of play in the tournament so far.
"It's going to hurt, that one," said Murdoch, who endured his own fair share of pain during three Olympic appearances as a player, including when he came up short in the men's final against the USA in Sochi in 2014.
"It was a really well controlled game and they were doing everything we asked of them. It's the best I've seen us play all week and for five ends we were clearly the better team.
"Unfortunately when it comes to the power-play, if you do have a couple of small errors you can easily lose three. And that was actually the deciding point of the game."
It was a missed take-out attempt by Dodds in the seventh that proved pivotal, handing the Norwegians the three, and when Britain could only respond with one in the next, Norway took the 'hammer' - or final stone advantage - into the last end.
"In mixed doubles it's had to say you're in control the entire game - mixed doubles is so hard to keep on top of things," said Mouat. "The fact that we tried to make her play a really tough one in the last shows how resilient we were.
"We were basically trying to guard everything and they were trying to make the hits, so with Jen's last we just needed to put it in a really good position to force Kristen into playing a really tough draw.
"But unfortunately we over-swept it a bit and gave her too much backing to play her draw, so we're a little bit disappointed with how we handled that."