The Scots converted all five of their penalties, with Kenny McLean scoring the winner, after David Marshall saved Eran Zahavi's first spot-kick for Israel.
It ended an evening which had largely lacked quality, but was packed with drama. Scott McTominay should have scored just before half-time when he headed a corner wide when unmarked, while Liam Cooper's header in the final minute of extra time hit the post, but this was a nervous encounter that deserved a shootout to decide it.
Israel failed to create many opportunities to trouble Marshall, but were more than a match for Scotland throughout, and were inches from a winner in extra time when Shon Weissman failed to get a touch on Hatem Elhamed's cross.
Steve Clarke's side will travel to face Serbia on November 12, with the winner earning a place in England's group at Euro 2020 - two of the group matches will be played at Hampden Park. It will be the first play-off final for Scotland since 2003, when the Netherlands won 6-1 over two legs to reach Euro 2004.
How Scotland scraped through to set up Serbia showdown
Given what was at stake for both teams, it was a huge shame that the effects of Covid-19 had torn a hole in both teams. Israel were without six players due to positive tests, while Scotland had been rocked 24 hours before kick-off when Stuart Armstrong tested positive, with Kieran Tierney and Ryan Christie forced to isolate due to being close contacts.
All three would have been in contention to start had they been available, and injuries to Scott McKenna and Liam Palmer meant Clarke was forced into personnel changes - whilst keeping the back three he had introduced in September. Manchester Utd midfielder McTominay once again lined up alongside Motherwell's Declan Gallagher and Liam Cooper, with Stephen O'Donnell at right wing-back; there was also a new forward pairing in Oli McBurnie and Lyndon Dykes.
Even in an empty stadium, there was an undeniable feeling of nervous tension given what was at stake, but with the enthusiastic support of their team-mates on the bench - who were brandishing megaphones - ringing around an empty Hampden, Israel settled quickly. Yet it was John McGinn who had the first effort on goal, shooting over wildly from 25 yards.
Captain Andy Robertson was next to try, curling a freekick just wide following a foul on Dykes at the edge of the area. While Scotland were looking more comfortable, Israel's pace on the break was a constant threat, with Shakhtar Donetsk forward Manor Solomon looking particularly threatening.
Of the three Celtic players on the pitch, two of them were Israeli; Nir Bitton, passed fit despite Neil Lennon's assertions that he wouldn't be available, and Hatem Elhamed. It was the latter who had Israel's first good chance, cutting inside O'Donnell's sliding challenge on the left edge of the box, only to be denied by a good block from his club teammate Callum McGregor.
Chances were few and far between, with both teams cancelling each other out in midfield, and the best opportunities seemed likely to come from set-pieces. Bitton nearly took advantage with a header from a freekick, but he put it well wide under pressure from Gallagher.
McTominay then spurned a golden opportunity to give Scotland the lead just before half-time, when he headed wide from Robertson's corner. Unmarked at the back post, it was a glaring miss, and left Clarke holding his head in his hands.
From another set-piece McTominay came close again shortly after the interval, but this time he didn't make a clean connection and the chance was gone. Israel countered - Golasa played a one-two with Zahavi before whipping a left-foot shot just over, with Marshall scrambling.
The game was gradually becoming more open, but also tense and fractious. Bitton was booked for showing his studs to McBurnie, minutes after a similar challenge on Ryan Jack, while McGinn was also cautioned for a foul on Bibras Natkho. The Scots were then screaming for a penalty after the ball deflected off the arm of Eitan Tibi, but nothing was given, despite the option of a VAR replay for the first time in a Scotland international.
McGinn looked Scotland's best hope of creating something from open play, and he spun away from Bitton before crossing for McBurnie, but the striker's knockdown couldn't find Dykes. McBurnie then just failed to latch onto Dykes' header across goal as Eli Dasa produced a last gasp clearance - but Scotland had still not tested Ofir Marciano.
As the game entered the final quarter, Clarke made changes, with Lawrence Shankland replacing McBurnie and Ryan Fraser on for Jack, but their introduction did little to affect the stilted nature of the game. With six minutes left, Gallagher rose highest in a crowded box but headed wide of the far post. As injury time began, those of a Scottish persuasion held their breath as Marshall was left flat-footed, when McTominay deflected Elhamed's curling shot just past the far post.
The game entered extra-time, the first for Scotland since losing a play-off to Czechoslovakia in 1961 for a place at the World Cup. Callum Paterson replaced Dykes, but neither team could find a way through in the first 15 minutes. Fraser's pace had added an extra element to Scotland's attacking play, and his cross was met on the volley by McGregor, but Bitton made a strong block in front of Marciano.
As the clock ticked past 10pm, the second half of extra-time began, and the Hampden clock stopped working, as if sensing what many in the stadium felt - that this game might remain goalless forevermore.
Israel were still pushing for a winner, and Eran Zahavi, who had been very quiet, found space in the box but thumped a drive over the bar. Then with four minutes left, Elhamed's teasing cross flashed across goal, and Weissman only needed to get a touch on it to divert it home, but he failed to connect, to the relief of the Scots.
In the 121st minute, Cooper met Robertson's corner, but his header clipped the outside of the post with Marciano struggling to cover. It was the last action as penalty kicks loomed - Scotland's first-ever shootout in men's international football.
McGinn squeezed the first Scottish penalty under former Hibernian team-mate Marciano, before Marshall produced a superb diving save to deny Zahavi. The next three kicks from both teams were perfect, leaving the door open for substitute McLean to send Marciano the wrong way, and book Scotland's flight to Serbia.
Clarke: We're developing
Scotland manager Steve Clarke on Sky Sports:
"At the moment you have to enjoy it, it's another step towards hopefully making a bit of history for the country, qualification is in our hands, but it's one we'll look forward to in Serbia.
To be honest, the game went how I expected, tense, nervy, not many shots on target. We had the best chance of the game, Scott McTominay's header in the first half, and we almost nicked it at the end from another set play.
"As a group players and squad I think we're developing, I can feel something building. It's another game unbeaten, you can feel it after the game, you can feel the elation. We put everything in and got our reward in the end.
"Tonight was not about digging out a result, it was about being careful not to concede in a game that was probably going to be decided by one goal. They blinked first and we won the shootout.
"We go there with every confidence, going there looking to make a little piece of history for Scottish football."
- This was Scotland's first goalless draw since November 2013 against USA, having gone 54 matches in all competitions without a stalemate prior to today.
- Scotland have gone six games without defeat in all competitions (W4 D2) for the first time since October 2017 under Gordon Strachan (a run of seven).
- Despite being 29 shots in this game only one landed on target (Eran Zahavi's attempt for Israel in the 72nd minute).
- Despite having 17 attempts, Scotland failed to land a single shot on target in a home game for the first time since September 2013 against Belgium.
Scotland now host Slovakia in the UEFA Nations League on Sunday evening, live on Sky Sports Main Event at 7.45pm, and they then host Czech Republic on Wednesday evening in the Nations League, live on Sky Sports at 7.45pm.
They'll then face Serbia on November 12, following their 2-1 extra-time win over Norway on Thursday.
Source: Sky Sports / Charles Paterson