The Spaniard stopped Maxence Lacroix's spot kick to seal the win 8-7 on penalties after the teams ended 1-1 at Emirates Stadium on Tuesday.
It was a tough night for Lacroix whose own goal in the 80th minute had given Arsenal the lead before Marc Guehi equalised for Palace deep into stoppage time.
Manchester City play defending champions Newcastle in the other last-four clash.
The Gunners took the lead when Palace failed to clear Bukayo Saka's corner. Lacroix knocked the ball over his line and shook his head in disbelief.
In the fifth minute of stoppage time, Jefferson Lerma headed down Adam Wharton's free kick for Guehi to sweep it into the goal.
The penalty takers were perfect up until Lacroix's effort. Arrizabalaga went to his right and stopped it.
Arsenal manager Mikel Arteta was pleased with the resilience shown by his team to hold their nerve despite the late goal they conceded.
"The margins should have been bigger and we will work on that," he told Sky Sports, referring to this game and their narrow 1-0 win over Everton at the weekend.
"We generated a lot (of chances). We should have scored many more goals than the one that we scored. Overall, another victory, I'm really happy."
Palace manager Oliver Glasner said he was happy with his team's second-half showing.
"Walter Benitez kept us in the game, the second half was much more even," he said, adding his players must now switch off for a day or two to enjoy Christmas with their families after a hectic schedule.
Title-chasing Arsenal host Brighton & Hove Albion in the Premier League on Saturday, while eighth-placed Palace are at home to Tottenham Hotspur on Sunday afternoon.
