Gillingham keeper Jason Brown was his side's hero as he kept Rotherham at bay with a handful of impressive stops.
But then in an impressive second-half display Paul McLaren put the home side ahead, with Deon Burton and Martin Butler adding goals late on.
Early on Butler did well to set up leading scorer Burton but Brown saved his first-time shot.
Butler controlled well in the area in the 25th minute and sparked a snapshot from Lee Williamson and Brown made another sharp save.
Burton spun and fired a 25-yard effort wide after a clever move involving John Mullin and Butler.
And when Butler cut in from the left Brown dropped to hang on to the low shot.
Matthew Jarvis went close for Gillingham and ex-Rotherham striker Darren Byfield almost embarrassed his old side when he beat Shaun Barker and David Worrell, but hit his shot wide.
Danny Jackman came close giving Gillingham a first-half lead but glanced his header wide.
It took Rotherham 54 seconds of the second half before they turned their dominance into a goal.
A slick link-up between McLaren and Williamson produced an opening and McLaren found space and steered his shot past Brown.
Burton produced an emphatic finish in the 80th minute after Butler burst clear down the left and Butler claimed the last goal in the dying seconds.