But Broughton came off the bench in the second half and claimed his sixth goal of the season as Rotherham hit the visitors on the break in the 68th minute.
The red card came when keeper David Stockdale, on loan from Fulham, rolled the ball to Joseph, but as he dithered Vale striker Kyle Perry nipped in and stole the ball.
Desperate Joseph tripped Perry as he threatened to dart in on goal from 25 yards out and was immediately sent off.
Vale failed to make the most of the extra man as the home side defended stubbornly.
Perry did find space on the edge of the area and drilled a low shot that failed to trouble the keeper before David Howland hit an ambitious effort.
Rotherham were grateful to Stockdale earlier when Dave Brammer hit a stinging 25-yard shot that had the keeper diving full stretch to turn it round the post.
As the home side struggled to find a way through Vale's tightly packed rearguard Jamie Green did well to cut back a dangerous ball after a quick break down the left, but Richie Barker missed his chance to add the finishing touch.
The home side had to rely on break away opportunities and when Dale Tonge burst down the right, Danny Harrison might have done better but fired wide.
The breakthrough came after 67 minutes when a long clearance from Ian Sharps bounced over the defence and Broughton, brought on to replace Barker, hoisted a deft ball into the net as Joe Anyon raced off his line.
Vale sub Danny Glover came within a whisker of an equaliser when a clearance ricocheted off him and inches wide of the Millers goal.
