Major Conservative donor and former party treasurer Lord Ashcroft warned that the rows over Europe, gay marriage and "loongate" risk pitching the Tories into a "spiral of irrelevance".
Conservative co-chairman Lord Feldman will face questions about allegations he described grassroots activists as "swivel-eyed loons" who forced MPs to adopt hardline stances, at a meeting of the party's board.
The peer, a close friend of David Cameron, has strenuously denied making the comments after internet speculation linked him to the alleged remarks.
The row over the disputed remarks came as the Tory high command faced attacks from within the party over the Government's plans to legalise gay marriages.
Writing on the ConservativeHome website, Lord Ashcroft said: "This sort of thing - the habit of distracting ourselves from what matters - has got to stop.
"The Conservatives were elected, albeit in insufficient numbers, to do a job. We hope to be elected in two years' time in rather greater numbers to implement a Tory manifesto in full. But we don't look as if either of those things is our priority. We need to pull ourselves out of what threatens to become a spiral of irrelevance."
Lord Ashcroft said it would be "worse than regrettable" if anyone at the top of the party had insulted grassroots activists. But he said the row over the alleged comments highlighted a problem facing the Conservatives.
"For all its virtues, the Conservative membership does not look the same as the rest of Britain," said Lord Ashcroft. "This is an observation, and a pretty inescapable one, not a criticism.
"Tory members are, by and large, older and better off than voters as a whole, and their political priorities do not always match those of the people we need to convince if we are to win."
Activists need to decide whether they want the party to be "a vehicle for their views" or are prepared to "grin and bear" policies designed to broaden its appeal, while the leadership need to show that the Tory agenda extends beyond Europe and austerity, wrote Lord Ashcroft.
The Marriage (Same Sex Couples) Bill returns to the Commons today for two further days of debate today. MPs led by Conservative Tim Loughton have tabled an amendment that would expand civil partnerships to include heterosexual couples.
But the government and gay rights campaigners have warned the amendment is not about creating equality and instead designed to delay or even wreck the entire bill.
The gay marriage debate has split the Conservative Party. Writing on The Huffington Post UK today, Tory MP David Burrowes said: "The Bill creates a state orthodoxy which gives succour to the intolerant baying crowd or the politically correct council in discriminating against supporters of traditional marriage."
But Burrowes' Conservative colleague Damian Collins used a HuffPost UK blog to argue: "everyone should be equal in the eyes of the law, and that the lifelong commitment people make to each other when they make their marriage vows means something profound to them, and benefits society as a whole".