Mertens battled back from 3-0 down to draw level with the reigning and seven-time world champion, before O'Sullivan clinched victory with a break of 63 to set up a meeting with Thailand's Dechawat Poomjaeng.
O'Sullivan said of Mertens on Eurosport: "He's a great cueist, he hits the ball well. I like his game. He's going to be around. The sky's the limit for him really.
"I think he has a very good snooker brain, sees the shot, plays the shot. Great lad, and just plays the game nice."
Things also went down to a deciding frame for Judd Trump as he edged Jackson Page 4-3, with his 21-year-old opponent failing to take a chance in the seventh frame to win the match.
Trump said in quotes on wst.tv: "I didn't want to be going home today. It could have been easier because, at 2-0 (to Trump), Jackson butchered a shot but incredibly he got away with it, and after that the match changed.
"He made a mess of certain shots today but overall he has improved massively and he looks a tournament winner to me."
Mark Selby was another big name that emerged triumphant from a tight encounter, getting past Noppon Saengkham 4-3.
Defending champion Neil Robertson made breaks of 110, 122, 67 and 88 en route to a 4-0 victory over Andrew Pagett.
Monday's other results included Mark Allen beating Mitchell Mann 4-1, Mark Williams defeating Matthew Stevens 4-3, 4-0 wins for John Higgins over Gerard Greene and Kyren Wilson against Reanne Evans, and Shaun Murphy posting a 4-2 victory over Anthony Hamilton.
Stuart Bingham, the 2015 world champion, was knocked out, losing 4-1 to Thepchaiya Un-Nooh.
Yan Bingtao had been scheduled to face Ashley Hugill, but after the Chinese world number 16 was suspended from the World Snooker Tour amid an ongoing investigation into match-fixing, Hugill was awarded a walkover.