The Gunners were seeking a club-record-equalling 11th home top-flight victory in a row, but Unai Emery's decision to make mass changes to his starting lineup backfired.
Yet more sloppy defending gifted James McArthur a third goal for the Eagles, which would prove to be the winner as Pierre-Emerick Aubameyang pulled one back late on for Arsenal, who missed out on the chance to move into third place.
Emery promised to make changes on the back of Thursday's victory at Napoli, which sealed Arsenal's place in the last four of the Europa League, and so it proved as just four players survived from that match.
One of the new faces to come in was Konstantinos Mavropanos, who gave away a needless free kick for the game's opening goal 17 minutes in.
Arsenal's best chance up until that point fell to Ozil inside the opening minute, but the German midfielder could not find a way past Vicente Guaita from a good position.
The Gunners soon took control of the match and racked up a couple more openings, with Shkodran Mustafi failing to convert a flick-on at the back post and Alexandre Lacazette dragging wide after losing his balance.
Arsenal also had a penalty shout rejected when Sead Kolasinac's cross hit Aaron Wan-Bissaka on the arm, though they were perhaps fortunate not to be further behind at half time as Cheikhou Kouyate was twice denied from the angle by a double Leno save.
Benteke then squandered a massive chance to double his and his side's tally when firing a first-time shot over the bar at the end of a quick counter led by Zaha.
Those changes instantly paid off as Iwobi picked out Lacazette, who in turn played a reverse pass into the feet of Ozil to hammer past Guaita at his near post with a trademark shot-into-the-floor finish.
Despite dominating possession, the best Arsenal could muster over the next 15 minutes was a long-range drive from Kolasinac, aimed right at Guaita.
With an hour played, and against the run of play, Palace were back in front as Benteke flicked on a 50-yard free kick and Zaha, having got the wrong side of Mustafi, converted past Leno.
Scott Dann fired at Leno from the game's next big opportunity, but the Eagles did not have to wait long for a third goal as Milivojevic's outswinging corner was headed goalwards by Dann and helped on its way from four yards by McArthur.
A 19th Premier League goal of the season for Aubameyang 13 minutes from time restored some hope for the hosts - the Gabonese drifting in from the right and firing a snapshot into the bottom corner after earning some luck with a favourable bounce of the ball.
Arsenal had another late penalty claim rejected for Dann's tug on Lacazette's arm, and Iwobi shot too close to Guaita when teed up just inside the box, meaning just a second home league defeat of the season for the Gunners.
ARSENAL (3-5-1-1): Leno; Mustafi, Koscielny, Mavropanos (Iwobi 46'); Jenkinson (Maitland-Niles 46'), Guendouzi, Elneny (Torreira 68'), Kolasinac; Ozil; Lacazette, Aubameyang
CRYSTAL PALACE (4-5-1): Guaita; Ward, Kelly, Dann, Wan-Bissaka; Meyer (Townsend 80'), McArthur, Milivojevic, Kouyate, Zaha; Benteke (Ayew 78')