Protesters have gathered outside the Department for Work and Pensions demanding the sacking of welfare minister Lord Freud, who was recorded suggesting that disabled people were "not worth" the full minimum wage.
This comes as nearly 24,000 people have signed a petition calling on David Cameron to sack the controversial peer.
The "#FreudMustGo" protests will pile pressure on the prime minister, who has signalled that he has full confidence in Lord Freud after he issued a "full and unreserved apology" for his comments.
Labour demanded the Conservative minister's resignation over the remark, which sparked anger from disability charities and trade unions.
But Cameron's official spokesman said the Prime Minister wants "all Government ministers to be getting on with implementing policy".
Cameron flatly disowned the peer's remarks when ambushed by Labour leader Ed Miliband at Prime Minister's Questions and later ordered him to apologise.
Lord Freud has been withdrawn from frontbench duties in the House of Lords, where he had been scheduled to reply to a question and represent the government in a debate.
The Department for Work and Pensions said the peer "isn't available" but gave no further details of why he was being replaced by colleagues.
Lord Freud was recorded at a fringe meeting of last month's Conservative conference responding to Tunbridge Wells councillor David Scott, who expressed concern that some "mentally-damaged individuals" who want to work are unable to do so because employers were unwilling to pay them the £6.50-an-hour minimum wage.
The minister replied: "You make a really good point about the disabled ... There is a group - and I know exactly who you mean - where actually, as you say, they're not worth the full wage and actually I'm going to go and think about that particular issue, whether there is something we can do nationally, and without distorting the whole thing, which actually if someone wants to work for £2 an hour, and it's working, can we actually..."
In a statement later issued by Lord Freud, he said: "I would like to offer a full and unreserved apology. I was foolish to accept the premise of the question. To be clear, all disabled people should be paid at least the minimum wage, without exception, and I accept that it is offensive to suggest anything else.
"I care passionately about disabled people. I am proud to have played a full part in a Government that is fully committed to helping disabled people overcome the many barriers they face in finding employment."
He added: "I am profoundly sorry for any offence I have caused to any disabled people."
The Tory peer has caused his own share of controversies as a coalition minister, after attacking people for pursuing a "lifestyle" on benefits,, insisting there was no link between benefit cuts and an increased use of food banks, telling families hit by the bedroom tax can "go out to work or use a sofa bed", and likening benefit claimants to corpses.
The Huffington Post UK also revealed that the Department for Work and Pensions spent nearly £75,000 on media training for officials, including Lord Freud, leading one MP to quip: "I hope he kept the receipt."