Postecoglou's winless run since he took over 22 days ago continued and the Forest fans vented their frustration, with chants of "sacked in the morning" aimed at the Australian alongside songs of support for former boss Nuno Espirito Santo.
The result meant Postecoglou became the first permanent Forest boss to be winless in his first six games in 100 years.
The home supporters had waited with anticipation for their first taste of home European action since March 1996, but Ousmane Diao's opener killed the buzz inside the stadium.
Dan Ndoye equalised for Forest in the 22nd minute, only for Mads Bech to put the visitors in front again two minutes later.
Midtjylland had their backs to the wall in the second period and Forest twice thought they levelled through substitute Chris Wood, but on both occasions the assistant referee's offside flag cut celebrations short.
Instead Midtjylland caught Forest on the counter-attack and added a third through Valdemar Andreasen, sparking the chants against Postecoglou, with Wood's late penalty unable to stop the boos from ringing around the City Ground.
In Rotterdam, Emi Buendia and John McGinn struck second-half goals as Aston Villa beat Feyenoord 2-0 to make it back-to-back Europa League wins.
Villa, who beat Bologna 1-0 in their tournament opener last week, held their nerve at De Kuip after coming under heavy first-half pressure, with Buendia and then McGinn firing killer blows for Unai Emery's side.
Back at the venue where they sealed the club's greatest triumph by beating Bayern Munich to lift the European Cup in 1982, Villa notched their third win in a week to put their dismal start to the season behind them.
Villa were forced into a late change before kick-off, with goalkeeper Emi Martinez withdrawn through a calf injury and replaced by summer signing Marco Bizot.
In some of the other results in the competition, Braga beat Celtic 2-0, Lyon defeated Braga by the same margin and Sturm Graz toppled Rangers 2-1 to put under-fire boss Russell Martin on the back foot.
VfB Stuttgart missed a penalty in a painful 2-0 Europa League defeat in Switzerland to Basel with goals from Albian Ajeti (3rd) and Xherdan Shaqiri (84th), while Porto beat Crvena Zvezda 2-1.
Albian Ajeti in the third and Xherdan Shaqiri in the 84th gave Basel their first points and denied Stuttgart a second victory, after their Ermedin Demirovic missed from the spot in the 36th.
In the Conference, Crystal Palace extended their long sequence of games without a loss with a 2-0 win over Ukrainian champions Dynamo Kiev in Poland.
Daniel Munoz scored Palace's first-ever away goal in Europe just after the half-hour mark and second-half substitute Eddie Nketiah also struck, both from Yeremy Pino left-wing crosses, in the 58th minute.
The Eagles are currently second in the Premier League and the only unbeaten team in the competition under the tutelage of manager Olivier Glasner.
Palace have gone a club-record 19 matches without defeat and meet Everton this weekend.
