After a bright start, and forcing several corners, the Millers should have taken an early lead when Dale Tonge crossed for Kevin Ellison but his mishit shot missed the target.
Nicky Law was next to trouble Shakers goalkeeper Wayne Brown but as he raced into the area his shot was well saved.
Hot shot Adam Le Fondre was the talk of the opening period, as he saw two glaring chances go begging.
The striker found acres of space at the back post from Micky Cummins' deep cross but somehow scuffed his shot from six yards on the angle and Brown pounced to gather.
And Le Fondre was in the thick of the action again moments later as his header from Ellison's whipped free kick was parried onto the bar by Brown from point blank range.
Just before the break the game livened up once more, as firstly Brown was substituted due to injury and Ryan Lowe had the home sides best chance of the first half but fired wide from 20 yards when it looked easier to hit the target.
And the Millers got the lead they deserved just before the break as Ellison netted directly from a corner for the second time this season, as the ball sped into the net past replacement 'keeper Belford.
Rotherham's joy was shortlived though and missing Nick Fenton through suspension told as frailties began to appear at the back after the break.
Richie Baker picked up a loose ball and after running at the heart of the defence he fed Lowe who ran clear unmarked to slide the ball past Andy Warrington.
The main talking point came after sustained home pressure on 65 minutes when Ian Sharps' clean tackle on Lowe was bizarrely punished by referee Webb, who after speaking with his assistant, who never flagged, awarded a penalty.
However justice was done when Stephen Dawson's kick was saved by Warrington and the Bury midfielder blazed the rebound over the crossbar.
Millers hearts were broken with seven minutes to go when Baker pounced on a deflection in the box to poke home the winner.
Rotherham threw bodies forward and should have snatched an injury time point, but Belford went from zero to hero as he somehow tipped away Sharps' point blank header.