Children in care in England are forced to wait an average of 20 months to move in with adoptive parents, according to new figures.
Children's Minister Tim Loughton said the first-ever local authority scorecards were a "trigger for urgent, detailed discussions" to speed up the process.
It is part of an action plan for adoption which includes proposals to reduce the length of the approval process for would-be adopters to six months.
Council leaders and children's services professionals condemned the scorecards and warned they have the potential to cause "unnecessary and avoidable concern in communities where there shouldn't be any".
The figures show that 80 local authorities met the interim thresholds of 21 months from entering care to adoption and matching a child to a family within seven months of a court order.
But 72 councils did not meet one or both of these thresholds which will be lowered to 14 months and four months respectively within four years.
Children in care in Hackney, east London wait the longest in the country to move in with adoptive parents, at two years and nine months on average.
Merton, south London has the second longest time, at two years and eight months, and Liverpool is third at two-and-a-half years.
Loughton told BBC Radio 4's Today programme there had been a need to "overhaul" the whole adoption system, "not just tinker around".
He said the delay was "the most damaging thing for a child who comes into care, before they eventually find a safe, stable, loving home, with an adoptive family if that's the best destination for them".
Loughton said the adoption scorecards could "cast a spotlight" on the extent of the job to do in adoption and was "not about targets".
He added: "I am not going to introduce targets, that can have some very perverse consequences as the previous Government found.
"What the adoption scorecard does is to give complete transparency of information, it shows those aspects of the whole system that are working well in certain authorities and those that aren't."
He earlier described the scorecards as "not the be-all and end-all" but said more areas needed to strike a better balance between quality placements and the risk of long-term damage to children by leaving them with uncertain futures.
The scorecards feature three key indicators relating to authorities' performance on adoption:
The average time it takes for a child to be moved in with an adoptive family
The proportion of children waiting longer than they should, including those still in care
The average time it takes an authority to match a child with a family after a court has decided that adoption is the right course.
But council leaders and children's services professionals said local authorities could not risk shifting their focus from the quality of placements to speed.
The Local Government Association, Association of Directors of Children's Services and the Society of Local Authority Chief Executives released a joint statement which said: "Councils are passionate about helping children and take their responsibilities towards those in their care extremely seriously.
"The adoption scorecards have the potential to cause unnecessary and avoidable concern in communities where there shouldn't be any, and may put prospective adopters off. Children waiting for adoption will not benefit from government struggling to get its act together.
"The data fails to provide a sound basis for comparison across local authority areas. For example, one council's Ofsted-rated outstanding adoption service looks like a poor performer in the scorecard. This is simply not credible."