Tim Loughton said Sarah Teather failed to support married couples while in government as she "doesn't really believe in family".
The jibe was reportedly made at the Conservative Renewal conference in Windsor over the weekend. Loughton and Miss Teather were colleagues at the Department for Education before both being fired in the 2012 reshuffle.
According to the Daily Telegraph, he told activists that policies such as a married couples' tax break had been frustrated by her opposition.
"The person who was actually in charge of family policy amongst the ministerial team at the DfE was Sarah Teather. Which was a bit difficult because she doesn't really believe in family," he said.
"She certainly didn't produce one of her own. So it became a bit of a family-free zone. I think that is a huge disappointment."
Loughton, a father-of-three, is said to have since described the remark as "throwaway" and admitted it was unfair.
But a senior Lib Dem source demanded that he apologise. "I know he's angry about being sacked, but he needs to gather whatever dignity he has left and shut up," the source added.
Labour equalities spokeswoman Kate Green said: "It is highly inappropriate to be putting people's personal circumstances and private choices into public play in that way. It is a pretty unpleasant intrusion and I hope Tim Loughton will apologise.
"It is not the case that people who do not have children of their own do not care about families. Children are everybody's future; society as a whole has a stake in their well-being.
"My concern is not whether Sarah Teather has children of her own but the fact that she was part of a Government that introduced many policies that were extremely harmful to children."
In a series of posts on Twitter this evening, Loughton denied any "personal" attack on Miss Teather.
He said he would apologise for the comments as reported, but insisted they were "certainly not intended or indeed said in that way".
"My criticism was lack of Lib Dem family policy not of Sarah Teather personally who I always respected," he wrote."If some journo distorting my comments on LDem family policy in2 personal attack on Sarah Teather v sorry but not intended & not fair.
"I will of course apologise to Sarah for the comments that have been reported and were certainly not intended or indeed said in that way."
Liberal Democrat equalities minister Jo Swinson said: "Tim Loughton is talking absolute rubbish. Sarah Teather was a brilliant minister.
"She was pivotal in ending the disgraceful practice of child detention for immigration purposes, she successfully won more money for schools to help the poorest children, and her work to create a better system of support for children with special educational needs has been widely praised. She has a ministerial record to be proud of.
"Using someone's lack of children as an insult is shameful, crass and, frankly, disgusting. It reflects far more on him than on Sarah."Suggest a correction