Norwich kicked off attacking the Barclay End goal, but showed little offensive flair in the first 45 minutes.
John Mullin produced a powerful volley after 11 minutes which was blocked by Malky Mackay before it could trouble keeper Robert Green.
Seven minutes later Phil Mulryne had a similar chance but his effort went harmlessly wide.
The game was 25 minutes old before the first of Norwich's five corners, while Rotherham managed only one in the whole game.
The Millers went ahead in the 33rd minute when Darren Garner's 30-yard right-foot free-kick curled inside Green's right-hand upright.
There were chances for Norwich either side of half time, Gary Holt failing to make contact with a through ball and Mark Rivers getting in a shot on the turn which flashed past a post.
Craig Fleming was booked for a foul seven minutes into the second half, before Mike Pollitt was at last forced to make a save, getting down well to hold Mulryne's low drive.
On 55 minutes Nigel Worthington made a double substitution taking off Chris Llewellyn and young Ian Henderson to bring on David Neilsen and Paul McVeigh.
It was McVeigh who scored within minutes curling a 20-yard right-foot shot inside Pollitt's left-hand post.
Garner was booked for body checking the lively Neilsen on the hour as the home side at last began to look the more likely to snatch a win.
Mackay headed wide from a Rivers corner, then there was a double substitution by the visitors. Former Canary Mark Robins and Paul Warne replaced Darren Byfield and Chris Sedgwick.
Zema Abbey made way for veteran striker Iwan Roberts with a quarter of an hour left to play but he was unable to find the target on this occasion.