Fifth seed Thiem won 7-6 (7-3) 7-6 (7-4) 4-6 7-6 (8-6) at Melbourne Park.
The 26-year-old, beaten by Nadal in the past two French Open finals, missed two match points before taking a third to win in four hours and 10 minutes.
He will now face Germany's Alexander Zverev in Friday's last-four match.
It will be a first semi-final in Melbourne for both men and the winner will face either Serbia's defending champion Novak Djokovic or Swiss third seed Roger Federer in Sunday's final.
Thiem missed the chance to serve out the match at 5-4 in the fourth, becoming jittery with three unforced errors and a double fault as Nadal levelled despite doing little to earn it.
But Thiem regrouped to win the third tie-break of a thrilling contest against the 19-time Grand Slam champion.
"I was holding my serve great and then it came to 5-4," Thiem said.
"That was a special situation for me, serving against Rafa for a place in the Australian Open semi-finals for the first time.
"It was a mentally tough situation but I turned it around again in the tie-breaker."
Thiem raised both arms in the air when Nadal hit a forehand into the net on the third match point, clutching and shaking his head in what was likely to be a mixture of relief and disbelief.e
Posts
