Jeff Goulding got Cheltenham going after just four minutes as he slotted home on the half volley from a left wing cross, before Le Fondre levelled with his first - a sweet hit from the edge of the box after a touch from Tom Elliott.
Cheltenham then took over as the Millers chased shadows from the next 15 minutes or so, with Wes Thomas (28) tapping in.
Shaun Jeffers (36) then netted after Exodus Geohaghon's sloppy pass and sold Dean Holden short as Rotherham's defence lost their shape and their way.
But Danny Harrison got Rotherham back in it four minutes before the break from the edge of the area, curling in right footed after Nicky Lw's run infield from the right.
Le Fondre then levelled two miinutes after the break, converting a rebounded penalty after Tom Elliott was shoved over just inside the area.
Alfie then bagged his second as the Millers took the lead, stooping to head home after Ryan Cresswell headed down a free kick.
Cresswell (55) then gave United some breathing space with a towering header and Le Fondre added another, slipping the ball into the corner on the net right footed from 18 yards with still 25 minutes to play.
Bizarrely, the visitors then hit the foot of the upright and Thomas saw a penalty saved by Andy Warrington, with Keith Lowe heading a consolation right at the death to top a frantic afternoon.