Lord Patten said Farage and Ukip are "probably a balloon which is deflating", while Miliband is "highly intelligent" and a "good debater".
This flies in the face of conventional wisdom in Westminster - which says Farage's persona plays well with audiences while Miliband's failure to come across as authentic is damaging.
Lord Patten, who chaired the Tory party from 1990 to 1992 and stood down as chair of the BBC Trust last year, told Radio 4's Week in Westminster: "I wouldn't be worried about the Farage factor; I'd be much more worried about the platform it gives Ed Miliband."
He added Miliband could come across "a lot better" than expected during the campaign for May's general election.
It comes as David Cameron continues to refuse to participate in any pre-election debates to which the Green Party is not invited, while Ukip is.
The left-wing Greens are seen as a bigger threat to Labour than the Tories.
"I've been very clear you can't have some minor parties without other minor parties," Cameron said.
"I think the point I made about these debates being outside the election is being borne out by the fact some new outlets find it hard to talk about anything else."
Shadow Chancellor Ed Balls has called Cameron "yellow" for his stance.
During a visit to the US, Balls said: "I think it would be impossible for any incumbent US president to have any prospect of being elected unless they were in the debate.
"Any politician trying to become president or vice president who said in an American context they weren't up for a debate because they didn't like the terms would immediately in American terms be labelled as 'yellow'.
"And the Americans will never elect a yellow president.
"So I think, when people look across the water at our debate, I think they find it quite baffling that, when it comes to the Prime Ministerial debates, we seem to have an uncertainty about whether our PM wants to have a debate or not."
He added George Osborne would be "up" for these debates.
He said: "Whatever I think of George Osborne - and we disagree on lots and lots of different things - I've never thought he was somebody who tried to duck debates. I've never thought of him as being somebody who was a coward.
"It's not going to go away because people want a debate. I don't really know what's going on."