Brown was called to the Welsh Under 21 squad training camp on Monday but they refused to let him participate in the Millers game on Tuesday night with Sheffield United, even though their was no actual midweek International fixture.
With Scott short of fit defenders, he told Millersplayer he was unhappy with the situation.
"He got called up for a game that's at the weekend so they wouldn't allow him to play for us on Tuesday. It's madness", said Scott.
"For a young player who's got the opportunity to play first team football and to enhance their career, they are being pulled out of it which is limiting the games they are going to play for their club.
"We've asked for a little bit of leniency and a little bit of common sense but it wasn`t forthcoming."
Scott added; "We've covered nearly everyone in the country regarding who might be available for loan but it's frustrating, we will continue to look at bringing in players who will give us the cover and to push the ones who are in the side."