Miller resigned as culture secretary this morning after succumbing to a week of Labour criticism, internal Conservative Party arguments and an increasingly bitter battle with the press.
The prime minister told the Commons on Wednesday it was right to "allow her to make her apology and continue with her job" rather than ditch his cabinet minister at the "first sign of trouble".
Speaking during prime minister's questions, Cameron said he knew public "anger is still very raw" over the expenses scandal. but he said when ministers faced accusations it would be unfair "just to instantly remove them rather than give them a chance to clear their name".
Ed Miliband said the fact Cameron had initially wanted Miller to continue in her job showed he had failed to recognise what she did wrong and warned that "undermined trust not only in his government but in politics".
The Labour leader added: "If it had happened in any other business there would have been no question about them staying in their job." Miliband said Cameron's decision to stick by Miller showed he had only accepted her resignation due to the "bad press" and was comfortable with Miller's "bad conduct".
Cameron also dodged questions as to whether Miller had been pressured by Downing Street or others to resign. He said she "took her own decision and communicated that decision in her letter".
This morning, ahead of PMQs, Cameron announced Treasury minister Sajid Javid had been promoted to replace Miller as culture secretary.
Treasury minister Nicky Morgan, has been promoted from economic secretary to the Treasury to fill Javid's old job. Morgan has also been appointed minister for women and will attend the cabinet in that role.
The appointment of Javid, who was seen as likely to receive a promotion in the next reshuffle, caught many Westminster observes off guard as it had been predicted Cameron would want to replace Miller with another woman.
The resignation of Miller and appointment of Javid means there are only now three women who are full members of the cabinet. A fact immediately seized upon by Labour. Shadow home secretary Yvette Cooper tweeted soon after the changes were announced: "Another Cameron (mini) reshuffle, & result is even fewer women in full Cabinet positions. What is David Cameron's problem with women?"