The home side took the lead through Deon Burton but then an own goal from Colin Murdock quickly followed by a second from Matt Harrold was enough to give Yeovil the points.
Rotherham made a bright start and Paul Evans flashed an early warning at Yeovil but saw his low drive blocked in a crowded area.
John Mullin fed David Worrell and when his testing cross broke to Lee Williamson the midfielder's blistering shot was high over the bar.
And when a chance dropped for Williamson again he dipped a clever shot from 25 yards that was close to beating keeper Stephen Collis.
Rotherham profited from their early possession when Burton grabbed his first goal for a month with a cool finish to put the home side in front.
Williamson was the inspiration with a clever flick over a crowd of players from the right and Burton controlled on his chest before crashing the ball past Collis from close range.
Yeovil, who had threatened on the break, almost levelled immediately after the goal when Chris Cohen found space but his Neil Cutler held on to his stinging drive.
On 31 minutes a stroke of luck cancelled out Rotherham's advantage when Phil Jevons found space on the left and curled a shot that looped past Cutler off the head of Rotherham defender Murdock.
Boosted by the goal and four minutes later the visitors went in front. Kevin Amankwaah crossed and the ball broke for Harrold who rifled his low shot wide of Cutler.
Rotherham produced a spirited second half and looked desperately for an equaliser but were kept at bay by determined Yeovil.
Restricted to long-range efforts, Rotherham's best opportunity came when Evans struck a fierce 77th minute free-kick which was just off target.