In a letter to the Sunday Telegraph, the Conservative Grassroots group expressed a "deep concern" about "the negative effect of the gay marriage Bill on both Conservative Party morale and electoral appeal."
The letter called on peers to reject the the Marriage (Same Sex Couples) Bill, which arrives in the Lords on Monday for the start of what is expected to be a stormy passage through the upper chamber.
It survived a Commons backlash when 130 Tory backbenchers opposed the move.
In the letter, Robert Woollard, chairman of Conservative Grassroots, wrote: "It (allowing gay marriage) is alienating much of our core support while failing to attract new voters with under two years to go before the general election."
He went on: "Long-serving party members - many of whom have had the responsibility of bringing up children themselves - believe that the family lies at the heart of Conservative values. The golden inheritance of every previous
generation, that has been lovingly handed down to us, is now being smashed on the anvil of 'equality and fairness'."
Meanwhile, former Tory Cabinet minister Lord Mawhinney said the advice the Prime Minister was getting was not "politically astute" and that Mr Cameron himself said before the election that he would not allow gay marriage.
He urged the PM to prioritise the economy and education.
He told Sky News's Murnaghan programme: "If you go right back to your programme or network three days before the last election the Prime Minister said that he had no plans to do this.
"I assumed like other people, I believed him."
He went on: "So at a time when the Government should be focusing all its attention on the economy, which is what will decide the next election, and perhaps on education where Michael Gove is doing a great job, we have this distraction following Lords reform distraction, following Alternative Vote distraction.
"This is not a high priority for the people of the country."
Lord Mawhinney said if Mr Cameron had a credible adviser alongside him with "the feel, the sense of the party", the
Tories performance in the polls would improve.
"It's hard to feel that the advice the Prime Minister is getting is as politically astute as it ought to be," he said.
"That probably means somebody alongside him who has the influence, who has the public standing and credibility and who has the feel, the sense of the party, and were that to happen and were Mr Cameron willingly to accept advice from
such a person, then the position of the party in the public opinion polls would probably improve."