With the match seemingly heading for a draw, the hosts made the breakthrough two minutes into added time when Oztumer scored direct from a corner.
Chris Forrester was sent off for protesting before Morris forced home a second in the 97th minute.
The result moves Jon Whitney's men up one place to 11th, now four points off the play-offs.
In a lively start, Peterborough threatened first from a rapid third-minute counter-attack but a combination of Neil Etheridge and the Saddlers defence managed to scramble the ball to safety.
The hosts responded with Kieron Morris, starting in place of the injured Flo Cuvelier, firing over from 25 yards after excellent hold-up play from Simeon Jackson.
Posh went close on 16 minutes as Junior Morais whipped a pacy 30-yard free-kick just wide of Etheridge’s right-hand upright.
The Saddlers - targeting a fourth successive home win - had a good chance of their own on 25 minutes.
Morris’ super raking pass picked out the run of Jason McCarthy, who reached the byline and crossed low to Jackson, but his near-post effort was brilliantly saved by Luke McGee.
McGee’s opposite number, Etheridge, was then called upon at the other end, the Philippines international beating away Craig Mackail-Smith’s rasping drive from an acute angle.
Keepers continued to come out on top, with McGee reacting smartly to claim Jackson’s looping header after a McCarthy cross wasn’t dealt with.
In first-half injury-time, McCarthy - a constant outlet down the right – latched onto Oztumer’s pass and turned Paul Taylor inside out before stinging the palms of McGee with a powerful low drive.
Half-time: Walsall 0-0 Peterborough
United had the first chance of the second half, Morais cutting inside onto his left foot and unleashing a low 18-yard strike which Etheridge kept out one handed at his near post.
Despite scoring nine goals in the previous three home games, the Saddlers were struggling to create any real opportunities.
The lively McCarthy had an effort charged down while Morris dragged a shot well wide from outside the box.
Jon Whitney replaced Jackson with Amadou Bakayoko on 74 minutes in an attempt to spark his side into life.
Posh captain Forrester then had a long-range drive held by Etheridge to his left, before Morris fired over from 20 yards.
The Saddlers pushed for a late winner, with Oztumer, signed from Posh in the summer, having an effort smothered by McGee after McCarthy’s shot deflected into his path.
But from the resulting corner, Oztumer whipped the ball into the near post, and somehow, it managed to find its way over the line.
Peterborough’s players were furious – claiming Bakayoko had pushed McGee – with Forrester being sent off for protesting.
And Walsall made the points safe in the seventh minute of added time, Morris squeezing the ball under McGee after substitute George Dobson’s initial effort was blocked.
Full-time: Walsall 2-0 Peterborough
Team: Etheridge, Preston, O’Connor, O’Connell, McCarthy, Chambers, Osbourne (Dobson 67’), Morris, Edwards, Oztumer, Jackson (Bakayoko 74’).
Subs not used: MacGillivray, Laird, Randall, Roberts, Moussa.
Attendance: 4,759 (571 Peterborough supporters).