The third division side went behind after just 60 seconds but gave Ronnie Moore's side problems with ex-Rotherham youngster David Merris in sparkling form.
Rotherham had a dream start when John Mullin's corner was met by centre-back Chris Swailes with a powerful header that hit defender Darren Edmondson on the line and screwed off him and into the net.
The home side should have stretched their advantage in the 23rd minute when Alan Lee's pace took him into the danger area.
Lee produced a clever clipped shot but Mark Ovendale made an acrobatic stop to turn the effort away and when Andy Monkhouse followed up his tame effort was deflected out for a corner.
But York were level after 32 minutes when Merris pounced on a free-kick from player-boss Chris Brass and smashed it past Mike Pollitt.
Merris, who was rejected by his hometown club as a teenager, was the villain when he gave away a penalty under pressure from Lee.
But the striker, who could be the subject of a £1million offer to Cardiff, fluffed the spot-kick and Ovendale made a good save.
The home side went in front again when winger Chris Sedgwick burst down the right and lashed a stinging shot past Ovendale just before half time.
Rotherham carved out more openings in the second half and a tremendous reverse pass from Lee sent in Darren Byfield, but his ball across the goal was smashed wide by Sedgwick.
And Brass had to be alert when Lee headed goalwards and the Minstermen's manager headed clear.
Another goal-line clearance from Lee Bullock denied Rotherham after 65 minutes when Chris Swailes headed goalwards.
Rotherham: Pollitt, Shaun Barker, Minto, Swailes, Branston, Talbot, Monkhouse, Sedgwick, Mullin, Byfield, Lee.
Subs: Jones, Hurst, Warne, Richard Barker, Baudet.
York: Ovendale, Edmondson, Brass, Hope, Bullock, Ward, Wood, Dunning, Nogan, Wise, Merris.
Subs: Cooper, Brackstone, Downes, Stewart, Reid.
Referee: Phil Prosser (W Yorkshire).
Attendance: 2,919