Set 170 runs for victory after winning the toss, Australia slumped to a dismal 4-29 inside the powerplay and gave Zimbabwe, 11th on the ICC rankings, reason to dream.
Glenn Maxwell (31 off 32 balls) and top-scorer Matthew Renshaw (65 off 43 balls) spearheaded the rescue mission with a 77-run stand for the fifth wicket in pursuit of Zimbabwe's 2-169.
But when Maxwell chopped on and last recognised batter Marcus Stoinis (6) holed out, Zimbabwe were on their way to dismissing Australia for 146 with three balls left on Friday.
The shock loss in their second World Cup match marked Australia's first T20I defeat to Zimbabwe since 2007.
Zimbabwe did not even qualify for the last edition of the tournament.
It also leaves Australia fighting to progress out of the group stage and beating co-hosts Sri Lanka in their next game is now non-negotiable.
But even with a win in Kandy on Tuesday, Australia could be left to rely on net run rate to qualify for the Super Eight phase.
The bloodshed began in the second over of Australia's run chase as Josh Inglis (8) holed out on the first ball that towering Blessing Muzarabani (4-17) bowled.
Cameron Green and a newly-fit Tim David were both caught behind for second-ball ducks to leave Australia 3-25.
Still deputising for injured Mitch Marsh as captain, Travis Head (17) could consider himself unlucky chopping onto his stumps.
But it nevertheless marked only the second time in history Australia had lost four wickets inside the powerplay at a T20 World Cup.
Batting without a helmet, Maxwell shook off his disastrous recent form and showed flashes of his flamboyant best before also chopping on.
Renshaw, meanwhile, found joy down the ground and notably hit Wellington Masakadza for back-to-back boundaries in the 14th over as Australia staged their comeback.
The No.6 raised his bat for a first T20I half-century in his fifth appearance.
But Australia's key man skied one in the penultimate over facing Muzarabani, who finished with career-best figures after being named player of the match in Zimbabwe's last-start win over Oman.
The wicket of Renshaw left the bowlers with 31 runs required with eight balls left.
The Zimbabweans did it all without their vice-captain Brendan Taylor, out with a hamstring injury, and with captain Sikandar Raza battling calf cramps throughout the fielding innings.
In a particular highlight of a great day's fielding for Zimbabwe, Tony Munyonga dived in the deep the for a superb catch that dismissed Ben Dwarshuis (6).
Earlier, 22-year-old opener Brian Bennett (64 not out) top-scored for Zimbabwe as wickets proved elusive for Australia.
Stoinis had the first breakthrough dismissing Bennett's opening partner Tadiwanashe Marumani (35), caught behind before sending an injury scare through an already depleted Australian camp.
Cameron Green (1-6) struck to dismiss Ryan Burl (35) with the first ball after Stoinis left the field with a hand injury, suffered when Burl blated a full toss back at him.
The key men from the last-start win over Ireland, Nathan Ellis (0-34) and Adam Zampa (0-31), went wicketless but at least helped Australia restrict Zimbabwe to four boundaries from the last four overs.
15°C