Carney said banking culture needed a "fundamentally important" change and should avoid the type of attitude that led to the mis-selling of banking products.
He told BBC Radio 4's Today programme: "The cultural issue is fundamentally important. There has to be a change in the culture of these institutions. It's something I've spoken about in the past when I was in Canada, and it applies absolutely here as well.
"I think finance can absolutely play a socially useful and an economically useful function but what it needs in order to do so, the focus has to be, of the financier, the people working in the banking system, has to be on the real economy, what it does for businesses making investment, what ultimately it means for jobs in the economy.
"And it's the loss of that focus, it's finance that becomes disconnected from the economy, from society, finance that only talks to itself and deals with each other, that becomes socially useless."
"One of my other responsibilities is chairing the Financial Stability Board and a lot of what we're doing there internationally is to strip out that type of behaviour."
Pressed on whether bankers should sell a product that makes a profit, even if they knew it was bad for the consumer, Carney replied: "Absolutely you shouldn't sell it. Let me be absolutely clear as well, the type of behaviour you're talking about is conduct behaviour.
"But let me say as governor of the Bank of England, and this is something that is shared by my colleagues, it's that attitude which indicates even if we're not directly responsible for that behaviour, it's that attitude in institutions that undercuts their effectiveness, is bad for the system, and to the extent that with our powers we can use them, we work to snuff them out."
Carney's tone has provoked derision from financial commentators. Ann Pettifor of Prime Economics told the Huffington Post UK: "It is not enough to issue exhortations to the finance sector. The Governor of the Bank of England, who acts as guardian of the nation's finances, must regulate and discipline the financial system in order to create a culture which is socially useful."
Carney's call for banks to be "socially useful" echoes the 2009 warning from Lord Turner, the former chair of the now defunct Financial Standards Authority, that much of the City of London's activity was "socially useless".
Lord Adair Turner, chair of the Financial Standards Authority, until its abolition in March 2013
The governor was speaking after unveiling the Bank's new policy of 'forward guidance' on Wednesday, which indicated that the Bank would keep interest rates at their historic low of 0.5% until 2016.
The Bank would only start to consider increasing interest rates if unemployment drops below the 7% margin, requiring an extra 750,000 jobs to be created to reduce it from its current 7.8% level and inflation to remain stable.
The governor said he had "tremendous sympathy" for savers who would be financially worse off by the Bank's announcement to keep interest rates low.
"We have tremendous sympathy for savers. They have done the right thing, they have set aside some money, they are earning a lower return than they would have expected when they put money aside but the best way to get interest rates back to a normal level is to have a strong economy," he said.
"Let me tell you, I first lived in this country in the 1980s. After that I lived in Japan and it was the start of life after the financial crisis and in Japan they made two mistakes."
"The first was they didn't fix the banks quickly enough - in the UK we are fixing the banks. Secondly, as their recoveries began, they pulled back on stimulus too early and those recoveries didn't gather life and as a consequence, almost a quarter-century later, interest rates are still at rock-bottom levels in Japan."
"We don't want to make those mistakes here in the UK. This is about ensuring that in the reasonable future, and I recognise that it's difficult for savers, but in the reasonable future we're delivering that strong economy that will deliver higher interest rates and jobs."
Prime Minister David Cameron said he was welcomed Carney's forward guidance as reinforcement of the coalition's economic strategy.
"What is good about what Mark Carney is saying is that he is effectively saying look, the Government is doing the right thing by taking difficult decisions to get the deficit down and therefore we can have an aggressive monetary policy until unemployment falls even further."
Cameron, however, didn't mention Carney's extra warning this morning that the UK was still a "crisis economy" and experiencing the "weakest recovery on record".
Meanwhile, the Bank’s Quarterly Inflation Report released yesterday included figures showing that Britons will be £100 billion poorer than they would have been if the 2008 financial crash had never happened.