The on-loan Manchester United goalkeeper kept out spot-kicks from Ruben Neves and Joe Hodge as Forest emerged 4-3 winners after it had ended 1-1 in normal time.
Willy Boly scored against his old club to put Forest ahead but Raul Jimenez levelled after the break in a bad-blooded encounter that saw a mass brawl on the pitch at the end of the penalty shootout, which is sure to land both clubs in hot water.
Forest are in the last four of a competition that they have special memories in, having won it four times, and now a return to Wembley is a possibility after they were drawn against Manchester United in the two-legged last-four tie.
Steve Cooper reverted to their strongest XI, having made 10 changes for the FA Cup defeat at Blackpool on Saturday, and their intent to try and progress was clear from the off.
They made a vibrant start with Morgan Gibbs-White cutting back for Gustavo Scarpa, but the Brazilian could not get his shot right and Wolves survived.
Forest took a deserved lead in the 18th minute as the goal was made and finished by two Molineux old boys.
Gibbs-White whipped in a dangerous corner to the near post, which was deflected on to the crossbar and Boly was on hand to poke home from close range.
Boly showed his respect for the club he played for almost 150 times in a five-year spell by refusing to celebrate, but the rest of the City Ground made up for it.
Henderson had been a spectator for the opening 35 minutes but the on-loan goalkeeper proved his worth with two saves in as many minutes to keep his side at bay.
First, he got down well to keep out Jonny's low effort and then produced a brilliant stop as he stayed big to beat away Jimenez's effort from 12 yards after Wolves had cut Forest open.
An even better save came four minutes later as Wolves began to threaten regularly.
They broke down the right and Jimenez sent in an inviting cross for Hee-Chan Hwang, but the South Korean's first-time effort was pawed away by Henderson.
Having weathered the storm before the break, Forest regained the ascendency after the restart and had chances to extend their lead.
Gibbs-White picked up a loose ball on the edge of the area and forced Jose Sa into a low save and then, after the former Wolves man jinked into the area, his ball back was met by Serge Aurier, whose shot was blocked.
Forest were made to regret not taking one of those chances as Wolves levelled out of the blue in the 64th minute.
They worked the ball out to the right and Matheus Cunha, who had only been on for six minutes, put a ball across goal that Jimenez could not miss.
Forest regrouped and were within inches of going back in front as an attempted pass inside the area ricocheted off Nelson Semedo and on to the post.
Wolves thought they should have had a penalty late on as Emmanuel Dennis tangled with Cunha, but referee Graham Scott waved away protests, which sparked a melee between both sides and earned visiting boss Julen Lopetegui a booking.
That meant the game ended up being decided from the spot as it went to a shoot-out.
Penalty specialist Henderson came to the fore as he made saves from Neves and Hodge to send his side through, before ugly scenes on the full-time whistle.