Robertson produced three century breaks against O'Connor in their last-four clash at Meadowbank Sports Centre in Edinburgh, but that was not enough against the world number 55.
O'Connor watched Robertson open with a score of 137 and after taking the next two frames could do nothing as the Australian made successive scores of 127 and 116 to lead 3-2.
Leicester's O'Connor then made a 137 to level it up at 3-3 and snatched the seventh frame with a nerveless 47 clearance, which proved decisive before sealing his win with a score of 71 in the final frame.
Earlier on Saturday, Gary Wilson reached the third ranking final of his career by defeating Thailand's Thepchaiya Un-Nooh 6-4.
World number 32 Wilson, who knocked Ronnie O'Sullivan out in the third round and beat namesake Kyren in the last 16, sealed his victory over Un-Nooh with his third century break of the match in the 10th frame.
Wilson moved into a 5-3 lead after successive scores of 122 and 130 and looked set to wrap it up in the ninth frame before missing an easy red to the middle.
Un-Nooh, who defeated Judd Trump 5-4 on a re-spotted black in his quarter-final, cleared up to steal the frame on the black and make it 5-4, but Wilson produced a match-winning score of 115 in the next.