Arsene Wenger's charges edged through 4-2 on aggregate thanks largely to a 3-0 victory in the first leg, as goals from Hosam Aiesh and Ken Sema punished an out-of-sorts Gunners side in the return fixture in north London.
Graham Potter's Ostersunds will head out having recorded victories against two former Champions League constants this term, having also defeated Galatasaray 2-0 in Sweden during the qualification rounds.
Ostersunds started the contest with real purpose and industry, which came in stark contrast to the Gunners, who barely got going inside the first 20 minutes.
The Swedish side soon sprung into life in the 22nd minute, when Saman Ghoddos produced a delightful, cross-field ball for Aiesh, who then saw his shot across the face of goal deflected on its way into the net by Calum Chambers.
Potter's charges delivered a second shock blow just two minutes later, this time as Ghoddos pounced on more slack Arsenal defending to find Sema, leaving the winger to produce some memorable skill inside the box before lashing a shot beyond David Ospina and into the far corner.
Wenger looked on with clear frustration from the bench as his side continued to toil in the first half, leaving Ostersunds to gain more confidence having settled into a comfortable lead heading into the break.
The Gunners might have grabbed a goal to settle any nerves just before the half-time whistle, only for skipper Jack Wilshere to fire over the bar from close range after profiting from Henrikh Mkhitaryan's free kick.
Wenger's men orchestrated the perfect response after the break, however, when Sead Kolasinac capitalised on some poor defending from Ronald Mukiibi to finish beyond Aly Keita and dampen any hopes of a memorable upset.
Osterunds continued to mount pressure despite the hammer-blow and they might have found a third goal had it not been for Ospina, who scrambled well off his line to deny an advancing Brwa Nouri.
The striker had another chance to earn a draw in the dying embers, but again the frontman was wasteful as he struck his effort straight into the body of Keita, whose save ensured a historic result that will live long in the memory of the travelling Ostersunds fans.
For Arsenal, defeat comes ahead of an important trip to Wembley in the League Cup final on Sunday, although the Gunners will head into that clash with Manchester City safe in the knowledge that their European aspirations are still intact.
ARSENAL (4-2-3-1): Ospina; Bellerin, Chambers, Holding, Kolasinac; Elneny, Maitland-Niles (Xhaka '46); Iwobi (Nelson '90), Wilshere (Willock '76), Mkhitaryan; Welbeck
OSTERSUNDS FK (4-4-2): Keita; Mukiibi (Islamovic '72), Papagiannopoulos, Pettersson, Widgren; Aiesh (Tekie '52), Edwards, Nouri, Sema; Hopcutt (Arhin '81), Ghoddos