Already assured of progress from Group E, the second-placed Red Devils arrived in Spain knowing they needed to win by a two-goal margin to usurp the first-placed hosts and progress straight to the round of 16.
Teenager Alejandro Garnacho got them off to a superb start but United lost their way after half-time, with Ten Hag's alterations ineffective as La Real dug deep in a 1-0 defeat that means they topped the pool.
The Reds finish runners-up and will face one of the sides dropping out of the Champions League, with Barcelona, Juventus and Ten Hag's former club Ajax among the possible opponents in Monday's play-off draw.
"You don't always get what you deserve - that's football," said the United boss, whose side lost September's Group E opener 1-0 after Real converted a controversially-awarded penalty.
"When (in the) first leg you get a penalty that isn't a penalty... today we win, so I'm happy with that. I'm happy with some other things like the performance from Garnacho.
"Donny (van de Beek) back in the team. I think we make a really good goal with Garnacho, some good chances as well in the first half.
"We win today but of course we are disappointed we didn't score two goals because that is what we needed."
Ten Hag struck a philosophical rather than frustrated tone following Thursday's defeat in the Basque country, where Garnacho once again impressed.
The 18-year-old received a standing ovation following his first start against Sheriff Tiraspol last Thursday and opened his United account with an excellent goal in San Sebastian.
Ten Hag was pleased with Garnacho - who overtook George Best as the club's youngest non-English goalscorer in major European competition - as the teenager stepped up in the absence of key players.
"We need options," the Dutchman told BT Sport. "We have problems there.
"Antony not available, Anthony Martial not available, Jadon Sancho not available, but I'm really happy with that performance and I hope he can keep the process going.
"He was a threat, he scored a goal, he was reliable defending so if you can keep this going I'm really happy."
Ten Hag praised Garnacho's "great finish" a week after revealing he had been irked by the teenager's attitude earlier in the season - a situation Bruno Fernandes expanded on in Spain.
"He was good but he knows we expect a lot from him," the Portuguese told BT Sport. "Obviously still really young, he's doing really well.
"At the beginning of the season he was not at his best. In the tour he didn't have the best attitude that he should have and that's why he didn't get his chances until now.
"He is getting his chances because he's training better, having a different attitude and deserving his chances."
United now turn their attentions to Sunday's Premier League trip to Aston Villa, which comes the day before a knockout round play-off draw that could see Ten Hag's side paired against a tough opponent.
"We don't need to look at that because our main goal was to go first, so we could avoid that (play-off)," Fernandes said when asked if he had been aware about the potential opponents.
"Now we have to play against one of them but that's no problem for us if we want to win the competition.
"We have to fight good teams in any moment of the competition so sooner or later we will be ready for that."