Mr Gove had wanted to bring in what he said were more rigorous exams in some core subjects from 2015, but faced criticism from MPs, teachers and supporters of the arts.
The change means plans for the new qualification, announced in September, are being shelved, while GCSEs are retained, despite having been previously condemned by the Education Secretary.
Labour's Shadow Education Secretary Stephen Twigg told MPs the announcement was a 'humiliating climb-down' and exam policy a 'total shambles'.
The new English Baccalaureate Certificate was billed by the Government as having a tougher syllabus, with exams at the end of the course in a return to an O-level style traditional qualification.
Mr Gove told MPs in the Commons today: "Last September we outlined plans for changes to GCSE qualifications designed to address the grade inflation, dumbing down and loss of rigour in those examinations.
"We have consulted on those proposals and there is now a consensus that the system needs to change.
"But one of the proposals I put forward was a bridge too far."
Specifically, he said that proposal had been to let just one exam board set a GCSE for each English Baccalaureate Certificate.
The idea behind this was to stop what he had called a 'race to the bottom' where he said exam boards might compete to offer easier qualifications.However, he also re-stated his belief in changes already being made to GCSEs, where there has been a switch to exams being taken at the end of two years rather than in stages, fewer re-sits and a reduced role for coursework.