There is a "strong case" for more elected leaders to drive growth and development in northern cities, such as Manchester, according to Osborne.
Speaking on BBC Radio 4's Today programme he said: "In this country we have some great cities, but none are on the scale of the global city that is London."
His comments came as he pressed the case for a new high-speed rail connection and better roads to link northern cities.
"We need to create a northern powerhouse, bring together cities, like Manchester, Leeds, Liverpool, Hull, that are physically quite close to each other, but don't have the transport links, don't cooperate in the way that you would see in a single global city."
He admitted that the economic imbalance towards the South East was partly the result of policy shortcomings. "I'm saying that our approach under governments of all colours has not been as successful as it might be.
"There has been one approach which is sort of just move some public sector jobs to different parts of our country and think that that is the core solution.
"There has also been an approach which says just leave it all to the market, and that hasn't really worked."
He suggested a mixture of both was needed to achieve the best results for the economy in the North. Mr Osborne said one of the main purposes of his speech was to "start a conversation", but added that work on better transport and policy-making connections between such cities - an idea first floated more than a decade ago - was "already quite well under way".
The Chancellor went on: "You also need to see strong city leadership."I think it helps to have a single individual who is, of course, democratically accountable, but also bringing together powers over planning, housing and transport."
"I think there is a strong case for elected mayors in places like Greater Manchester to make sure they have the same powers, the same clout, as the mayor of London does in our capital.
Boris hasn't always hit the headlines for the right reasons. Last year he was challenged on live television about the more unsavoury rumours surrounding him
Johnson was asked by BBC's Eddie Mair about a time he made up quotes while at the Times, and quizzed on why he allegedly "lied" about having an affair to the then Conservative leader Michael Howard.
He was also asked to talk about an infamous phone call between him and his former Eton friend Darius Guppy, recorded in 1990, where Guppy asks Johnson to find the address of a News of the World journalist who has upset him. Despite Guppy's apparent plans, the journalist was never attacked.
"What does that say about you?" Mair asked. "Making up quotes, lying to your party leader, being part of someone being assaulted – you're a nasty piece of work."
Referring to his affair with Petronella Wyatt, a fellow journalist which led to him being sacked as a shadow minister in 2004, Johnson said: "On that, I never had any conversation with Michael Howard about that matter. I don't propose to go into all that again. Why should I?"
The London Mayor repeatedly asked Mair to "talk about something else" and questioning if viewers were interested in his sacking from the Times, saying his writing had been "mild sandpapering" of the facts.