Mark Robins' side lived to fight another day in the lower reaches of League One, as Forest were repeatedly frustrated by goalkeeper Neil Cutler.
But Forest also still have their sights firmly set on second-placed City, who were defeated at Gillingham and now lead them by four points.
Forest almost drew first blood as David Prutton started and finished a move that saw him prod a James Perch cross narrowly wide and again as Grant Holt glanced a header narrowly off-target, in a frantic start to the match.
And the Forest stopper could do little as former Notts County man O'Grady hammered the visitors into the lead with a stunning 20-yard drive that fizzed into the top right corner of the net on ten minutes.
The end-to-end action continued as Nathan Tyson was denied by a fine save from Cutler, after being sent clear by Prutton. But it was almost 2-0 to Rotherham as a firm downward header from Ian Henderson forced a sprawling save from Smith at his near post.
Forest drew level in the 21st minute as Fleming's clumsy challenge on Tyson as he burst into the box from the left earned them a penalty, which Grant Holt duly dispatched confidently into the bottom right corner of the net.
Smith was at his best again to deny Rotherham as he flung himself low to his left to palm away a Delroy Facey header at the far post - but was also grateful to see Henderson fire badly wide when sent clear.
Chances were fewer in number after the interval, although O'Grady shot badly into the side netting only yards from goal for Rotherham and Kris Commons forced a spectacular stop from Cutler with a bending free-kick.
Grant Holt saw yet another header saved by Cutler, this time as he connected with a cross from substitute Julian Bennett.
While Commons saw David Worrell clear off the line after Cutler had saved bravely at Jack Lester's feet. But there was to be no way through for Forest.