David Miller struck a match-winning 59 from 46 deliveries to see his side over the line as they reached the required 134 to win with just two balls to spare.
South Africa had to recover from the loss of two quick wickets and 40 for three, with Aiden Markram's arrival at the crease helping push up the run rate after the drinks break.
Markram was crucially dropped by Virat Kohli in the 12th over when he was on 35 and went on to make 52 from 41 before being caught by Suryakumar Yadav.
Miller continued the charge for South Africa, scoring a boundary from the third ball of the final over to reduce the equation before following it up with a second four to win the game and secure a five-wicket win for his side.
Earlier, India - having elected to bat first - were 23 without loss before Lungi Ngidi caught and bowled India captain Rohit Sharma, a wicket which sparked a collapse.
India slid to 49 for five, with Virat Kohli, who had yet to be dismissed in the tournament, ousted for just 12 runs.
Suryakumar Yadav led India's recovery, scoring 68 off 40 before top-edging a Wayne Parnell delivery to Keshav Maharaj.
South Africa delivered an impressive bowling performance to limit India to a well-below par total of 133, with Ngidi the pick of the bowlers with four for 29, while Parnell took three wickets.