As world No.3 Coco Gauff survived a scare from fellow American Hailey Baptiste, and seventh seed Jasmine Paolini was sent packing by Iva Jovic, Mboko overcame a stern test from Danish 14th seed Clara Tauson.
The 19-year-old let three match points slip in the second set before prevailing 7-6 (7-5) 5-7 6-3 on Friday at John Cain Arena.
It meant 17th seed Mboko, who made the Adelaide International final this month, reached the fourth round at Melbourne Park - her best result at a slam.
"I think it was just self-belief at the end," Mboko said.
"I had so many chances in the second set, but I've got to give credit where it's due, she was playing incredible tennis, so I just had to lock in, in the important moments.
"This is my first time here, so I'm really happy to go through to the fourth round.
"You're in Australia, it's warm. It's not so warm back home, so it's nice to be here right now."
Mboko - one of five teenagers to reach the third round - said it would be "super cool" to face Sabalenka for the first time.
The world No.1 overcame a fierce challenge from Anastasia Potapova before her class won out in a 7-6 (7-4) 7-6 (9-7) victory on Rod Laver Arena.
Belarusian superstar Sabalenka, yet to drop a set this summer, had to grit her teeth and outfight Potapova - including saving four set points in the second set.
Sabalenka, 27, is determined to reclaim the Melbourne Park title after Madison Keys denied her a three-peat in last year's final.
"She played incredible tennis. I was always on the back foot," Sabalenka said.
"To be honest, there are days where you just have to be there, just have to fight and try your best to put the ball back on that side.
"I don't know what was the key. Even though I was emotionally all over the place, I was able to still fight for every point.
"I'm just super happy that I was able to get this win."
Paolini was beaten 6-2 7-6 (7-3) by Jovic, who registered her first victory over a top-10 opponent.
"It feels amazing," the American said.
"I've been wanting this one for a while now and working really hard for it.
"I've had a couple tough losses and I'm just so happy to get through that barrier and get the win today."
Jovic next meets Yulia Putintseva, who beat Zeynep Sonmez 6-3 6-7 (3-7) 6-3.
Gauff dropped the first set against Baptiste, but it seemed to steel the French Open champion, who fought back to win 3-6 6-0 6-3.
"Honestly, I didn't change too much," Gauff said of the turnaround.
"I think I tried to get more first serves in.
"She played so well in the first set, maybe a couple points go my way it could be a different outcome, but I wasn't too upset."
Gauff's fourth-round opponent is 19th seed Karolina Muchova, who crushed Magda Linette 6-1 6-1.
Ukrainian 12th seed Elina Svitolina advanced with a 7-6 (7-4) 6-3 win over Diana Shnaider, and will face eighth-seeded young gun Mirra Andreeva, who beat Elena-Gabriela Ruse 6-3 6-4.
