The Scots lost 3-0 to their opponents in their Group I opener in Nur-Sultan March, a result so devastating that it soon led to the departure of boss Alex McLeish, whose successor Steve Clarke watched attacker Baktiyor Zainutdinov stun the home side with a terrific opener in the 34th minute.
Scotland stepped up the pace after the break and midfielder John McGinn levelled three minutes into the second half with a deflected free-kick before captain Steven Naismith headed in a second in the 64th minute.
McGinn scored again in the 90th minute to keep Scotland third in their section with their third successive win, and although they have failed to qualify for next summer's finals through
the conventional method, they have the chance to get there through the play-offs.
Scotland will be at home on March 26 and will discover their opponents in Friday's draw. On this evidence, however, Scotland will have to improve markedly to reach the final.
Looking to build on the weekend's win against Cyprus, Clarke kept the same side which meant centre-back Declan Gallagher won his second cap, with Naismith leading the line again as captain.
However, any momentum carried over from Nicosia was not immediately apparent.
A sparse crowd on a cold, wet night shivered through a stodgy first half.
The home side were let off when Kazakhstan striker Aleksei Schetkin got on the end of Dmitri Shomko's low cross from the left but failed to make good contact from six yards out, with keeper David Marshall saving with ease.
McGinn fired wide of the target at the other end and there was little else of note until the 32nd minute when Ryan Christie's deflected shot was saved by keeper Dmytro Nepogodov for a corner
which he gathered from Naismith's header.
Just over a minute later Scotland found themselves behind.
After right-back Liam Palmer gave up possession, Zainutdinov took a pass from Schetkin all too easily and moved towards the Scotland penalty box before unleashing a drive from 22 yards which flew high past Marshall.
Ryan Jack had the chance to level five minutes later but drove straight at Nepogodov and Christie curled a free-kick from the edge of the box over the wall in added time before boos accompanied the interval whistle.
The second half was more encouraging for Clarke.
After Naismith was fouled by Yuri Pertsukh and Sergei Maliy at the edge of the box, McGinn's low free-kick from 20 yards was deflected off captain Bauyrzhan Islamkhan and into the net, wrong footing Nepogodov.
Suddenly there was some Scottish confidence.
Forrest and Naismith both hammered drives over the bar from inside the box as openings arose and then the Hearts attacker headed in from close range after Palmer's deflected cross from the by-line had looped high into the air, with McGinn challenging Nepogodov to clear the way.
Veteran Naismith was replaced by Oli Burke in the 77th minute and Christie made way for John Fleck before McGinn put some gloss on the result when he swept in a Greg Taylor cross in the dying stages, although massive improvement is needed if Scotland are to ultimately reach their first major finals since 1998.