Di Alexander - whose son is Chief Secretary to the Treasury - issued his criticism of the policy in the annual report of the housing association he chairs.
He wrote that the Lochaber Housing Association in Scotland had been "facing up to the considerable challenge presented by Welfare Reform changes".
He said: "The first of those to be implemented - the so-called Bedroom Tax - is particularly unfair in that it penalises both our tenants and ourselves for not being able to magic up a supply of smaller properties."
Government funding had been focused on providing homes with at least two bedrooms, he said, as "all parties hitherto agreed" they were best suited to the needs of the rural area.
Under the Government's welfare reform, social tenants deemed to have more bedrooms than they need have had their housing benefit reduced.
Ministers say private sector renters do not get spare rooms for free, and argue the change will save around £500 million annually.
But it has sparked protests across the country with opponents claiming it is forcing families into poverty and will increase the benefits bill by pushing people into the private sector.
Labour will press for a vote in the Commons on Tuesday on the measure.
Shadow chief secretary Rachel Reeves told the Sunday Mirror: "Even Danny Alexander's dad has joined millions of people who think the tax is unfair.
"But his son and the rest of David Cameron's Cabinet are more interested in giving tax cuts to millionaires than helping those struggling to make ends meet."