The League One side enjoyed a dream start after just 56 seconds when Semi Ajayi got in ahead of the advancing Town keeper Joel Coleman and headed in an Anthony Forde cross from the left.
Moments later Kasey Palmer looked to get the Terriers on level terms but Richard O’Donnell denied him with a decent save.
The hosts continued looking for a way through but the Millers comfortably held on until the interval.
The Terriers turned the game on its head in the opening nine minutes of the second half.
On 52 minutes substitute Phillip Billing scored from the penalty spot after Will Vaulks was adjudged to have pushed Laurent Depoitre in the area.
And just two minutes later Palmer picked out Joe Lolley inside the Rotherham area and he fired a left-foot shot into the far corner.
Kieffer Moore looked set to level on 62 minutes when he slid in to meet a Jon Taylor cross from the left, but he put his effort straight at Coleman from close range.
On 83 minutes Vaulks was denied an equaliser when he sent a header past Coleman only for the referee to blow for a foul on Michael Hefele during the build up.
In stoppage time, Coleman pulled off a good stop to block a Moore strike in the middle of the area and Jonson Clarke-Harris somehow sliced the loose ball wide with the goal at his mercy as the Terriers held on to reach round three.