England survived a batting collapse while chasing 204 for victory. They slipped from 176 for four in the 41st over to 196 for nine in the 46th before last-wicket pair Charlie Dean and Anya Shrubsole saw them home.
Bowler Kate Cross admitted England were still searching for the "complete performance" as they aim to reach the semi-finals. They face Pakistan in Christchurch on Thursday.
Cross said in an interview with the England and Wales Cricket Board: "Very-much relieved, yeah. We don't do things by halves do we?
"I think it's about parking this now and making sure we can move forward and do what we do well. We've not had that complete performance yet, so we'll look for that.
"We've been searching for that this whole World Cup. It's not like we're not trying. Sometimes I think cricket can be a bit strange.
"I guess the good side of it is you can take yourself out as a neutral and you've just played an incredible nail-biter in front of a packed-ish Eden Park."
The visitors won the toss and opted to bowl, with Manchester Originals duo Cross and Sophie Ecclestone each taking three wickets and Dean collecting a pair as New Zealand made 203 in 48.5 overs.
Nat Sciver top-scored for England with 61 while captain Heather Knight made 42 and following their late collapse veteran Shrubsole roared with delight after knocking the winning single.
Cross said she was confident Shrubsole and Dean would hold their nerve to get England over the line.
"Yeah, very confident," she added. "Anya's not a number 11, Charlie Dean is not a number 10. I think the fact we bat so deep is a massive positive for this team.
"When we get to crunch moments like that you know we've got players who have batted so well for us over the years.
"However, it's never nice when you know you're that last wicket. It's difficult to play your shots knowing that effectively you could lose it.
"So I'm so pleased for them both, so pleased for Anya, especially after the disappointment of the West Indies game when we couldn't quite get over the line. There's a lot of relief.
"We've obviously been playing must-win cricket for about a week now. We've got the win, which is the main thing."
England started the tournament poorly with defeats to Australia, the West Indies and South Africa before a change of fortunes with victories over India and now New Zealand.