In a letter to the Prime Minister, more than 30 present and former local party chairmen warned that Mr Cameron's backing for a change in the law had led to voters switching their support to Ukip.
They wrote that many of the lost supporters would not contemplate returning to the Conservative fold unless legislation for gay marriages was abandoned "or the party leadership changed".
The Marriage (Same Sex Couples) Bill returns to the Commons tomorrow for two days of debate, with many Tory MPs expected to oppose it on a free vote.
The letter to Mr Cameron was organised by the Grassroots Conservative group, whose chairman Bob Woollard said: "The Prime Minister's bizarre drive to ram this legislation through Parliament, without any democratic mandate and without the support of party members has been a disaster and has driven thousands of voters to Ukip.
"The marriage-based family is at the heart of Conservatism. This dilution and unraveling of marriage has demotivated many ordinary loyal Conservative Party members and has undermined their years of hard work for something they believed in. It makes winning the next election virtually impossible.
"It is vital that the PM and those around him concentrate on the issues that matter to voters - cleaning up the economic mess left by the previous government, tackling immigration and sorting out the Europe question. For the sake of our children they should also strengthen conventional marriage."
The letter says Mr Cameron's "refusal to listen to reason and grassroots opinion is causing many previously loyal Conservatives to leave the party; some are lost forever and many will not contemplate re-joining unless the Bill is abandoned or the party leadership changed".
It claims the legislation for same-sex unions had also undermined work to win support in ethnic minority groups "who cannot comprehend how a Conservative Prime Minister can be promoting a Bill that will redefine marriage in a way which is contrary to their religious and cultural beliefs and practices".
Hundreds of Muslim leaders have also attacked the gay marriage plans in an open letter to Mr Cameron, Labour leader Ed Miliband and Liberal Democrat leader Nick Clegg, the Deputy Prime Minister.
More than 500 community leaders and imams signed the letter which stated "marriage is a sacred contract between a man and a woman that cannot be redefined".
They wrote: "We believe that marriage between a man and a woman is the cornerstone of family life, the only institution within which to raise children.
"We are concerned that this radical change to the institution of marriage will impact what is taught in schools. Muslim teachers will be forced into the contradictory position of holding private beliefs, whilst teaching a new legal definition of marriage.
"Muslim parents will be robbed of their right to raise their children according to their beliefs, as gay relationships are taught as something normal to their primary-aged children."
Health Secretary Jeremy Hunt told BBC1's Andrew Marr Show he backed gay marriage, including in churches that were willing to conduct the ceremonies, but recognised it was a "difficult issue" for many people.
He said: "I personally support it, I support it because I believe in the institution of marriage and I think we should be encouraging people to make a lifelong commitment to each other. I think society is stronger if you do that.
"In my own case I got married in a church and not a register office because I happened to want to make my marriage vows in front of God.
"I think if gay people want to do that and if the church is willing to conduct that ceremony we shouldn't stand in their way."
He added: "It's one of those issues of conscience that is not really about party, I think there are people of all parties that find this a very difficult issue. But I think it is one we are right to surface and to have a proper debate about."
Tory former minister Nick Herbert said the move to support gay marriages reflected a change in society and would not do any harm.
He told Sky News' Murnaghan programme: "This isn't a Bill that will harm anyone, nobody has to enter a gay marriage, no church will be forced to conduct a gay marriage because the protections are there, the Church of England has said it's happy with the protections that have been given, they don't have to do it.
"So actually what harm is being done by this Bill? None. For those that are concerned about their own belief, they are entitled to their own belief but actually for the majority this is something that people are saying does reflect change in society."
Tory vice-chairman Bob Neill acknowledged there were "different views" within the Conservative Party on the issue but there was "respect" on both sides.
On the same programme shadow home secretary Yvette Cooper warned: "I think it's a real problem if this gets lost in the vortex of the Tory infighting that we've had over the last couple of weeks when actually it's a really positive Bill that we should all want to celebrate."