Goal ace Phil Jevons grabbed his 19th of the season from the penalty spot and Darren Way and Andrejs Stolcers grabbed two in two minutes in the dying minutes to earn a deserved victory.
Ronnie Moore's Yorkshire side, currently hurtling headlong towards relegation, could well be meeting up with promotion hopefuls in the league next season.
And the win was no more than Yeovil – roared on by 1,500 travelling fans – deserved after they rattled the hosts from the higher division.
The only surprise was that the side who chalked up four wins over Christmas and New Year had to rely on a goal from the penalty spot.
But the win was earned by Yeovil's high energy approach after they snapped at the heels of Rotherham and knocked them out of their stride.
Rotherham were struggling from the start as Yeovil, full of confidence and running, set the pace.
Giant Russian frontman Bartosz Tarachulski made his huge physical presence felt in the eighth minute with a bulldozing header from Lee Johnson's corner which dropped to Scott Guyett but his follow up shot hit Chris Swailes.
And in the stuttering start the Yorkshire side found it hard to create any promising openings as Yeovil's eager midfield got in first with a series of sharp challenges.
But Paul McLaren produced a dangerous ball driven in from the right but it was blocked by Guyett.
Latvian international Stolcers hoisted a wild ambitious shot high over the bar as Yeovil burst forward again.
Rotherham went close though midway through first half when a McLaren corner dropped kindly for Andy Monkhouse but his clever lob hit the bar .
Jevons saw an opportunity go begging when he curled a free-kick over the top from the edge of the area and minutes later his powerful run down the right set up Kevin Gall but his first-time effort was wide.
The first effort on target for either side arrived ten minutes into the second half when Jevons twisted and turned before hitting a 20-yarder that forced a save from Mike Pollitt.
Five minutes later Rotherham skipper Martin McIntosh felled Paul Terry in the area and Jevons stepped up confidently to strike home the resulting penalty.
It could have been worse for the home side when Stolcers hit a stunning 25-yarder five minutes later but it beat Pollitt and smacked against the bar.
Andy Monkhouse went close with a later long-range effort for Rotherham and with only seven minutes to go Yeovil keeper Chris Weale preserved the lead when he tipped over a cross-cum-shot from McIntosh and then made a superb double stop to parry a Monkhouse shot and then smother Martin Butler's follow up.
Then two goals in two minutes confirmed the win .
Three minutes from the end Way ghosted in behind the defence and slotted home a cool shot for the second.
And a minute later Stolcers smashed home a 15-yard shot to round off the victory.