David Cameron has sought to smooth relations with angry local Tories, insisting he would never have anyone close to him who "sneered" at them.
The Prime Minister emailed party members following weekend reports that a close ally dismissed grassroots activists as "mad, swivel-eyed loons".
He acknowledged differences of opinion but insisted his policy on Europe was "right for the country" and urged party members to unite around a "great fight" on issues such as the economy, schools and welfare.
Details of his plea for unity were released after the Conservative Party board rejected a call for an investigation into whether its chair, Tory co-chairman Lord Feldman, was behind the slur.
The peer, who was not named in newspaper reports, was forced to deny making the comments after intense internet speculation.
Brian Binley MP urged a probe at a meeting this afternoon but it was "pretty overwhelmingly rejected" and the board was "unanimous in its support for Andrew Feldman", he said afterwards.
Although his call for an investigation was rejected, "it was felt that there was a need to look at the narrowing of the gap between the party in the country and the leadership", the Northampton South MP said.
"I understand what David Cameron is saying about trying to reach out to more people than the Conservative Party, but he has to realise that he is the caretaker of the Conservative Party and not the proprietor of the Conservative Party."
Dissent over issues such as gay marriage and Europe have fuelled complaints from some senior Conservative volunteers that Mr Cameron is out of touch with the views of his party and is helping fuel an exodus to the UK Independence Party.
In his email, Mr Cameron said he wanted to send a "personal note" in response to the reported "loons" jibe.
Stressing his own background as a volunteer and local branch chairman, Mr Cameron said: "We have been together through good times and bad. This is more than a working relationship; it is a deep and lasting friendship."
He went on: "Above all, we Conservatives believe you change things not by criticising from your armchair but by getting out and doing.
"Time and again, Conservative activists like you stand for duty, decency and civic pride. That's why I am proud to lead this party. I am proud of what you do. And I would never have around me those who sneered or thought otherwise.
"We are a team, from the parish council to the local association to Parliament, and I never forget it.
"Does that mean we will agree on everything? Of course not. The Conservative Party has always been a broad church - one which contains different views and opinions - and we must remain so. But there is also much we must do together."
As well as hailing the Government's achievements, he told them, "we can be clear about where we are going, too.
"We are engaged in a great fight to rebalance our economy, to bring excellence back to our schools, to fix the welfare system. And yes, we have a policy on Europe that is right for our country.
"Amid all the debate, remember this: it is our Party that has committed to an in-out referendum on Europe by the end of 2017. Not Labour, not the Liberal Democrats, but the Conservatives who are committed to giving the British people their say.
"So to those reading this, here is my message: there will always be criticism from the sidelines. But we must remember what this Party has always been about: acting in the national interest. Our task is to clear up Labour's mess and make Britain stand tall again.
"We have a job to do for our country - and we must do it together."
Earlier, major Conservative donor and former party treasurer Lord Ashcroft warned that the rows over Europe, gay marriage and "loongate" risk pitching the Tories into a "spiral of irrelevance".
Lord Ashcroft said it would be "worse than regrettable" if anyone at the top of the party had insulted grassroots activists. But he said the row over the alleged comments highlighted a problem facing the Conservatives.
Activists need to decide whether they want the party to be "a vehicle for their views" or are prepared to "grin and bear" policies designed to broaden its appeal, while the leadership need to show that the Tory agenda extends beyond Europe and austerity, he said.
Mr Binley said there was a "growing gap" between the Prime Minister and his party.
He told BBC Radio 4's Today programme: "I think David Cameron was elected leader because we thought he was a winner. He almost did win the election at a time when tribal politics was decreasing."
Mr Binley said the Prime Minister had "done a few things that the party in the country overall didn't want him to do".
He added: "He may believe in them but we are talking about the gap between himself and the party. I think it is a growing gap, I think it can be put right and I think David Cameron is listening as proved by the fact that actions are now being taken."
The Tory MP said his local association was "really quite angry" about the "mad, swivel-eyed loons" reports.
"Clearly something was said and clearly we have got to have more control from the coterie around the leadership when they talk with the press because that's doing damage too."