Rotherham have appealed to the Football Association over the sending-off after club officials studied a video of the incident.
Assistant manager John Breckin said: "When the ball first went into the area, Todorov had a goal-scoring opportunity.
"But by the time there was physical contact with Swailes that chance had gone and he wasn't even looking to get a shot in.
"At first, we didn't think it was even a penalty but we are not arguing about that now, merely the fact that the offence doesn't warrant a red card."
If the Millers appeal fails they will be fined £10,000 for having six players either booked or sent off.
They were fined £5,000 for having seven players booked earlier in the season after the game with Sheffield Wednesday at Hillsborough.