Nick Gibb was forced to defend his government's academy plans after one headteacher's attack on the proposals was applauded by an audience of heads, teachers and education experts.
Executive headteacher David Kirk, who is in charge of two schools in Halifax, West Yorkshire, told the schools minister that Michael Gove's plans to deal with the 200 worst performing primary schools in England were unrealistic.
Kirk, who is supporting a school where around 70% of pupils are without English as a first language and have a range of problems, said some schools could not hope to exceed the criteria set out by the government.
"We will never beat these floor targets, ever," he told the North of England Education Conference.
Gibb was asked by delegates at the conference about his views on Gove's comments earlier this week in which he talked about "ideologues" and "enemies of promise" who were "happy with failure".
Gibb said: "It's reflecting the problems we've encountered with some local authorities who have not been as determined and ambitious for pupils in their area that we think they should have been.
"It's a minority but we think it's reprehensible that they're not taking action to tackle schools that for year after year have been letting down children in their school."
He said: "The overriding aim of this Government is to close that attainment gap between rich and poor and that will mean using harsh language, taking difficult decisions and challenging some of the vested interests that lie in some parts of the country that are not tackling some of these under-performing schools."
The minister told the conference in Leeds the government was right to challenge "vested interests" and said the autonomy offered by academy status was the best way to raise standards.
"The whole drive behind the academy programme is based on the evidence from around the world that it is the most effective way of achieving high standards in schools - to deliver autonomy and to give professionals that autonomy to run schools as they see fit."
In his speech to the conference, Gibb stressed the importance of the government's plan to convert the weakest 200 primary schools into academies.
He said: "If schools aren't making the right progress, and local authorities don't have a grip on the issue, we will be able to intervene to secure the best possible result for the children in those schools.
"So, by expanding the academies programme, increasing autonomy at school level and improving teacher training we want to drive up standards in schools right across the country."
But Labour's shadow education secretary Stephen Twigg criticised the programme saying the government was putting all their eggs "in one basket".
"Governance matters, but we shouldn't obsess about it. What matters most is what happens in the classroom - good leadership, discipline and quality teaching."