The government has defeated Labour's bid for a Leveson-style judge-led inquiry into the banking sector by 320 to 239, a majority of 81 against the proposal.
Instead a parliamentary inquiry made up of MPs and peers as proposed by David Cameron will be established to look into what went wrong at Barclays in the wake of the Libor rate-fixing scandal.
Speaking in the Commons before the vote, Ed Balls said there was "massive public anger" at the revelations of "lying and market manipulation" at Barclays had left "trust in our banks is in tatters".
"How we respond will have long term consequences for the future of banking and our economy," he said.
"Only and open and independent public inquiry not politicians investigating bankers can re-build trust."
Following the vote, Balls indicated Labour would co-operate with a parliamentary inquiry headed by the chair of the Commons Treasury committee, Andrew Tyrie.
Balls said that David Cameron and George Osborne had made a "very grave error of judgement" in refusing to back an independent inquiry.
He told MPs that Labour would continue to press for a wider Leveson-style inquiry into the culture and practises of the banks.
The debate in the Commons that preceded the vote was characterised by its highly partisan nature as Balls and Osborne traded insults over each other's conduct.
Balls had angrily demanded Osborne withdraw the "utterly baseless accusation" that he had intervened to try and manipulate the inter-bank Libor rate while he was in government.
Balls V Osborne
Osborne refused to withdraw or apologise for his comments to the Spectator and told MPs that Balls had "questions to answer" as it had been suggested Labour ministers had intervened to influence the Libor rate.
And Osborne said Balls should apologise himself for failing to regulate the banks properly while he was City minister.
"The idea that I am going to take lessons on integrity smeared his way through 13 years of Labour government who half the people who ever served with him think he was a disgrace in his post is another thing," he said.
"The Brownite cabal with all their tactics are alive and well in the Labour Party and have taken over the leadership."