The winning kick was scored by former Socceroo Kenny Dougall.
Aurelio Vidmar's men had a man advantage for more than 100 minutes, including added time, but were unable to make it count thanks to a stellar display from Buriram's ex-Fulham goalkeeper Neil Etheridge.
With the aggregate score ending 1-1 the Philippines international saved from Ryan Teague and Aziz Behich in the shoot-out, putting the Thai champions through to the quarter-finals.
Having snatched a late draw in the home first leg Melbourne were up against it at Buriram's Thunder Castle but their chances improved dramatically when Robert Zulj was sent off in the 21st minute.
The Austrian midfielder was dismissed for stomping on a prostrate Harrison Shillington. Zulj's brother Peter was lucky not to join him after putting Nat Atkinson in a neckhold in the ensuing scuffle.
Thereafter Melbourne dominated possession but found it difficult to penetrate a home defence well-marshalled by their former teammate Curtis Good.
When they did break through Etheridge denied both Daniel Arzani and Max Caputo in normal time and extra time, while Marcus Younis put a spectacular overhead kick just wide and Elbasan Rashani was also frustrated by Etheridge.
In the final minutes of extra time City's German Ferreyra was also dismissed after the VAR spotted an apparent retaliation on Ruben Sanchez.
That was Melbourne's only card of the night while Buriram had six yellows and a red.
They were fortunate that one of those yellows was not a red after Etheridge appeared to aim a kick towards Caputo in the 96th minute.
That proved significant.
While Caputo scored City's first spot kick Etheridge dived to his left to keep out Teague and Behich. While Younis then scored Patrick Beach had not been able to save any of Buriram's kicks so when Queenslander Dougall converted their fourth it was all over.
"Everything about the performance shows how together we are as a team," said Etheridge. "I'll take the plaudits for saving the penalties but for 100 minutes my team dug extremely deep to take it to the point where I was able to make those saves."
Meanwhile, Machida Zelvia advanced through a 1-0 win over Gangwon FC.
Hotaka Nakamura's 25th minute header confirmed Machida's progress after a 0-0 draw with Gangwon in the first leg last week.
Machida and Buriram are the first teams to secure their places in the quarter-finals, which will be played in Jeddah on a centralised basis along with the tournament's semi-finals and final from April 16-25.
Japan's Vissel Kobe take a 1-0 lead into the second leg of their meeting with FC Seoul on Wednesday while Malaysia's Johor Darul Ta'zim will defend a 3-1 advantage against hosts Sanfrecce Hiroshima.
Matches in the last 16 in the western half of the draw have been postponed due to the ongoing conflict in the Middle East.
with Reuters
