John Gummer, who sits as Lord Deben in the House of Lords, issued the warning after the Labour leader decided to confront accusations that he is too "weird" to be prime minister.
The former environment secretary and chairman of the Conservative Party told The Huffington Post on Tuesday that unless David Cameron and the Tory campaign machine treated Miliband with respect, the voters would not take the Conservatives seriously.
He also compared arguments over whether it was Labour or the Tories that started making personal attacks first to the situation in Gaza.
"One of the reasons people are turned off politics and politicians is name calling and personalities. It seems to me it treats politics rather like a celebrity gossip column," he said. "And I don't think that's what we should be talking about, we should be talking about ideas."
"Take him seriously, you look at his policies. That's the only way that you be taken seriously. You can not take a party seriously if it's main attack on its opponent is a personality attack," he said.
"It treats people as if they are ignorant. The British people don't like it. Politicians must realise it has an effect of denigrating all politics."
John Gummer says the Tories must stop the personal attacks on Miliband
In recent months Miliband has suffered sustained personal attacks from both Tories and internal Labour critics over his personality as well as his looks - with the infamous photo of him awkwardly eating a bacon sandwich is used to illustrate suggestions he is odd.
But Gummer sharply criticised the mockery. "Anyone can be photographed looking not at their best," he said. "The issue is not about what people look like, or whether they are telegenic. I am not interested in whether he can eat a bacon butty or not, that's unimportant. I don't want to have an augment saying, 'well he lives in this sort of house' or he has 'this sort of face'."
"I want a sane and grown-up politics. This seems to be far too close to American and Australian name calling, which is so unpleasant and so counterproductive," he said. "Unless we are prepared to argue the policies it seems to be we demean politics altogether.
"Having seen the disastrous situation in Australia where the whole thing has become a circus, or the US where it is almost ungovernable because of the personal attacks, I think are we going to get in to that situation unless we stand up and say we are not going to have it."
Asked why he thought the Conservative Party had decided to focus on Miliband personally, Gummer said he suspected it was because it was an "easy" tactic. "We can all make jokes about people," he said. "It is a sort of British habit which is always to pull people down instead of build them up."
Gummer added he wanted to see all parties "start setting higher standards in the way we deal with each other".
"I don't care who started it first, that's the sort of the argument why you can't find a solution in Gaza. You've got to say, I'm sorry I'm going to to break through this," he said.
Both Miliband and Cameron have hired campaign help from abroad. The prime minister brought in strategist Lynton Crosby. And the Labour leader has hired Obama campaign guru David Axelrod, among others, to help him win the 2015 election.
In the interview he also said ministers should drop personal attacks on their Labour opposition number. Gummer said he did not care if shadow business secretary Chuka Umunna was rich, "good on him, he's done well." And he said criticism of Labour shadow education secretary Tristram Hunt based on his wealth was not important. "I don't want to argue with Hunt on the basis he has a lot of money, if that's the case."
On Monday, Conservative Campaign HQ launched a new website, JustNotUpToIt.com, which details the numerous criticisms of the Labour leader has had from his own side.
Miliband has also been attacked by former Gordon Brown spin doctor, Damian McBride, for announcing policies that amount to "a great steaming pile of fudge".
Gummer was John Major's environment secretary from 1993-1997. In his long Commons career he also served as chairman of the Conservative Party in the mid-1980s. He was made a Tory peer in 2010 and picked as the chairman of the UK's independent Committee on Climate Change in 2012.