Goals from Oliver Rathbone and Shane Ferguson helped Matt Taylor's Millers to move five points clear of the relegation zone.
The Millers had won just once in 12 games since beating Sheffield United in November but came good with goals either side of half-time from Ollie Rathbone and Shane Ferguson.
They had to work hard for it as Joe Gelhardt's first goal for Sunderland since his January loan move from Leeds made a game of it, but Matt Taylor's side held on to move five points clear of the drop zone.
The Black Cats suffered their first defeat in six games, which damages their play-off hopes, though they are still within striking distance of the top six.
It was a quiet opening with little action at either end until Rathbone asserted himself on the game in style to give the Millers the lead in the 19th minute.
He picked up the ball from Tariqe Fosu in the middle of the pitch and drove forward before unleashing a low 20-yard shot into the bottom corner.
The lead was almost short-lived as Sunderland went straight up the other end and only a brilliant save from goalkeeper Viktor Johansson kept his side in front.
Alex Pritchard created space for himself on the angle of the penalty area by cutting inside and then sent in a curling shot that was heading for the top corner until Johansson intervened with a fingertip save.
The danger was not over for the Millers as Dan Ballard was left unmarked from the resulting corner, but he flashed a header wide.
The Millers wrestled control of the match and looked to have been denied a clear penalty when Luke O'Nien barged Jordan Hugill, but referee Chris Kavanagh waved away protests.
The home surge continued early into the second half and Rathbone tried to repeat the trick with a shot from the other flank, but Anthony Patterson got down well to save it.
Sunderland were having their best spell of the match when Rotherham doubled their lead in the 56th minute.
Fosu was again the creator as he tackled Dan Neil then set Ferguson free and the Northern Ireland international drilled a low 20-yard shot into the bottom corner.
The Black Cats were determined to make a game of it and reduced the deficit five minutes later.
O'Nien found space on the right and sent in a delicious cross that Gelhardt did not need asking twice to bury with a close-range header as he broke his duck.
That goal changed the dynamic of the game and the visitors threw everything in search of a leveller.
Jack Clarke saw an effort blocked and Wes Harding made a brilliant last-ditch tackle to deny Aji Alese.
Rotherham held firm and had a chance to kill the game in the final 10 minutes as they broke quickly but Conor Washington hit a post from just inside the area.
They could have done with that going in as Sunderland launched everything at them in the end and had two opportunities to claim a point, but Johansson was equal to Gelhardt's header and then produced a heroic save with his feet to deny the Leeds striker again.