The Austrian team had led the play-off 1-0 on the night and 3-2 on aggregate when Hearts had Tomas Magnusson sent off in the 76th minute.
Progress seemed assured for the hosts but Hearts forced extra-time through Landry Kabore before taking a shock lead via Calvin Miller's spectacular volley.
Tilio came on in the 102nd-minute as Rapid chased the equaliser, which they secured in the 119th minute. That meant penalties.
Vienna failed first but Oisin McEntee, with the chance to put Hearts through, had his kick saved. That left Tilio needing to score at 3-4 to push the spot-kicks into sudden death. But Hearts' Dutch keeper Beau Reus saved his kick.
Also missing out on Champions League action is former Adelaide United player Yaya Dukuly.
Dukuly started for Slovenian side Celje, but they went down 2-1 at home to Slovan Bratislava.
An Australia U23 representative, the Guinea-born striker won the penalty that led to Celje equalising before half-time only for Cristian Martinez to score in extra-time.
There was better fortune for twice-capped Socceroo Gianni Stensness, who helped Norwegian club Viking Stavanger reach the lucrative group stage of the Champions League after winning a two-leg play-off against Dinamo Zagreb.
Stensness will be joined by Socceroos Nestory Irankunda (Sporting CP) and Jordi Bos (Feyenoord) in the Champions League.
Viking, who also featured New Zealand's Joe Bell, won 3-1 on the night to progress 5-3 on aggregate.
Also progressing through the play-offs were AEK Athens and Fenerbahce, defeating Levski Sofia and Lyon respectively.
11°C