Weak maths students receive the worst teaching, an Ofsted report has revealed.
The publication, released on Tuesday, found pupils in lower ability sets and younger pupils received the weakest teaching, while many of the brightest pupils do not fulfil their potential when they get to secondary school.
Inspectors from the education watchdog saw teaching ranging from outstanding to inadequate within the same school.
But the Ofsted report Mathematics: made to measure also highlighted a dramatic increase in the take-up of the subject and further maths at A-level, and showed that the youngest children are doing better.
It found GCSE and A-level results continue to improve thanks to the sustained efforts of teachers and students.
Between January 2008 and July 2011, inspectors visited 160 primary and 160 secondary schools and observed more than 470 primary and 1,200 secondary mathematics lessons.
They judged that more than half the schools were outstanding or good in maths.
But there remained three key areas where maths teaching at English primary and secondary schools must be improved.
The survey found not enough is being done to help pupils who fall behind early to catch up.
The 10% who do not reach the expected standard at age seven doubles to 20% by age 11, and nearly doubles again by 16.
Figures showed that 37,000 of the highest attaining primary school pupils went on to achieve no better than a grade C at GCSE in maths last year.
Schools which routinely enter students early for GCSE mathematics are hindering their ability to reach the highest grades, the survey found.
Her Majesty's Chief Inspector, Sir Michael Wilshaw said: "I want all children to have the best education they can and mathematics is a fundamental part of that.
"It is essential for everyday life and understanding of our world.
"Too many pupils do not fulfil their potential, including many of the most able, and those who get off to a poor mathematical start or fall behind in their learning never catch up.
"We know it can be done.
"Over half of the schools visited in the survey were judged to be good or outstanding in mathematics, although even in these schools some inconsistencies in the quality of teaching need to be tackled."
He vowed Ofsted would help schools improve maths teaching and said more able pupils should be pushed to be ambitious.
Mike Ellicock, chief executive of the charity National Numeracy, which helps adults and children with maths, said: "We believe every child can become a numerate adult - with skilful teaching in school and encouragement at home.
"This impressive report draws attention to the vital importance of numeracy and we hope its findings will be acted upon.
"It is essential that we all recognise the need for better support for those who struggle to catch up - at school and post-school."
Professor Celia Hoyles, director of the National Centre for Excellence in the Teaching of Mathematics (NCETM), said: "A deep understanding of mathematics and of subject-specific pedagogy is crucial for teachers of mathematics.
"The NCETM welcomes the recommendations Ofsted have made, and looks forward to helping to embed them as part of the professional development of all mathematics teachers."