The veteran broadcaster defended the show's record for inviting women guests and pointed out the proportions were much higher than those for many areas of public life.
David's comments come after Dara turned on BBC1's political discussion show in a Radio Times interview after being asked about a recent decision to ban all-male line-ups for comedy panel shows.
In the interview he observed: "It's remarkable that this amount of time is spent debating women on comedy shows rather than, say, Question Time."
But stung by the suggestion that his show had done anything wrong, David said: "Seeking to deflect attention from the gender imbalance on comedy panel shows Dara O Briain suggests that critics look instead to Question Time. He has picked the wrong target.
"Of the 104 panellists in the present run 46 have been women (44%). Given that we invite guests prominent in public life this is not a bad record. Only 18% of the cabinet is female. Only 23% of MPs are female. Only 20% of FTSE board members are female. Our aim is to achieve a 50/50 balance but British public life has to catch up before we can.
"Mr O Briain - the mote is in our eye, the beam is in yours."
In the interview Dara had expressed his view that scrapping all-male comedy panels would mean viewers would see any female guest as a "token woman".
It stemmed from BBC director of television Danny Cohen pledging that "we're not going to have any more panel shows with no women on them. It's not acceptable."
But Dara suggested viewers will now view guests in a different light and said if changes were needed they should have "evolved".
"I wouldn't have announced it, is what I'd say. Because it means Katherine Ryan or Holly Walsh, who've been on millions of times, will suddenly look like the token woman," said the maths and theoretical physics graduate, soon to host a new series of School Of Hard Sums for the Dave channel.
"It would have been better if it had evolved without showing your workings, if you know what I mean. Legislating for a token woman isn't much help.
"A certain number of women want to go into comedy, and they should be cherished and nurtured, but you're not going to shift the fact that loads more men want to do it."
Defending the efforts to impose more of a gender balance, a BBC spokesman said: "We're confident that a wider range of voices and opinions will make for even better telly."