The Super Eagles had taken an early lead through Odion Ighalo's deflected effort in the 19th minute.
However, Cameroon turned the match around with two quick goals just before half-time.
Stephane Bahoken got on the end of Christian Bassogog's lofted cross to equalise in the 41st minute.
Clinton N'Jie then showed good strength to hold off a defender and slot in another to put the Indomitable Lions ahead.
At the start of the second half, Marseille forward N'Jie was denied by Nigeria keeper Daniel Akpeyi and Ighalo levelled things up at 2-2 just after the hour when he drilled in a volley.
In the 66th minute, Arsenal midfielder Alex Iwobi produced a neat first-time finish after running onto a pass from Ighalo inside the Cameroon box to edge Nigeria back in front.
Nigeria also saw late chances for Ighalo and Ahmed Musa, but had already done enough to make sure of a place in the last eight.