The world champions lost five wickets for 15 runs in less than 11 overs during a dramatic fifth and final day of the first Test.
With New Zealand all out for 169, the Tigers needed 40 runs to cause one of the biggest upsets in recent years.
Shadman Islam and Najmul Hossain Shanto were dismissed for the tourists but Mominul Haque and Mushfiqur Rahim completed the victory in 17 overs.
New Zealand slipped from 136 for two to 136 for five on the fourth evening and the Bangladesh bowlers continued to run riot the next morning.
Ebadat Hossain and Taskin Ahmed snared two scalps each, the former completing figures of six for 46, before Mehidy Hasan Miraz wrapped up the innings following a stunning boundary catch from substitute fielder Taijul Islam.
For Bangladesh, this was their first win away in New Zealand in any format of the game. The Tigers had gone 32 consecutive games without a win.
The Black Caps, meanwhile, went 17 home Tests without defeat - including eight home series wins. What's more, they hadn't lost to Bangladesh in 15 head-to-head Test encounters.