Many PE lessons are failing to improve pupils' fitness, while not enough youngsters are playing competitive sport to a high level, inspectors warned on Thursday.
In a new report, Ofsted raised concerns that many schools are failing to push their sportiest pupils, or help those that are overweight.
It warned that in some PE lessons there is not enough physical strenuous activity, with pupils spending too much time listening to teachers.
Overall, PE lessons are not up to scratch in around a third of primary schools and about a quarter of secondaries, the inspectorate said.
The report, based on inspections of PE in schools over the last four years, concludes that in general the subject is "in good health", with significant investment in the last decade.
But it warns that in more than a quarter of schools, PE teaching did not improve pupils' physical fitness.
Ofsted chief inspector Sir Michael Wilshaw said that PE was part of a child's entitlement to a good education, and that there are some areas for improvement.
"In particular, we found there often wasn't enough physical, strenuous activity in PE lessons. Some teachers talked for too long and pupils were not provided with enough activity to enable them to learn or practise their skills.
"In many of the schools visited, the more able pupils were not challenged sufficiently because teachers' expectations of them were too low."
In a number of primary schools, youngsters were subjected to PE classes with warm-ups that were too short and too easy, followed by "long periods of inactivity" while the teacher talked, the report said.
"Pupils were not challenged to warm up vigorously or build stamina and strength by participating in sustained periods of physical activity," it said.
"They were often prevented from exercising for extended periods because teachers interrupted their learning or took too long to introduce new tasks."
Ofsted also suggested that the most able youngsters were taught the same lessons as their classmates, and given tasks that were "frequently too easy for them."
And it warned that a fifth of primary schools were not making sure pupils could swim at least 25 metres by the time they left.
In secondary schools, sporty pupils were not given enough time to practise and achieve their best, the report said, and only a minority of schools played competitive sport to very high levels.
In general, "Too much teacher-talk and regular interruptions to record information and observe others performing prevented them from remaining physically active throughout lessons," the report said.
Very few schools had set up programmes to meet the needs of youngsters who were overweight or obese, the report added.
Sir Michael said that schools with good PE lessons provided an "ever increasing range of extra-curricular and traditional activities", helping pupils to achieve more.
But he raised concerns about competitive sport in schools.
"Our report found that only a minority of schools play competitive sport to a very high level."
Sir Michael said he is commissioning a fresh report which will compare competitive sport in state schools to that in the private sector.
School sport has become a political hot potato since the Olympics and Paralympics last summer, with calls to ensure that the Games intended legacy "inspire a generation", is fulfilled.
Ofsted's report says that the Department for Education should build on improvements already made to PE and "harness the interest and momentum" generated by the Games to create a new national strategy for school sport.
A new PE national curriculum, published last week, says that between ages five and seven, pupils should be taught to master basic movements such as running, jumping, throwing and catching, take part in team games and perform simple dances.
Between seven and 11, youngsters should play competitive games like football, netball, rounders, cricket and hockey, as well as learn to swim at least 25 metres.
And in secondary schools, students should "use a range of tactics and strategies to overcome opponents in face-to-face competition through team and individual games", develop their technique, take part in outdoor and adventurous activities and compare their performances with previous ones to achieve personal bests.
The new curriculum also says that pupils should take part in competitive sports and activities outside school through community links or sports clubs.
A Government spokeswoman said: "We want all children to be given the opportunities they need to be fit and healthy.
"The draft PE curriculum published last week is designed to put competitive sport back at the heart of school life and end the damaging 'prizes for all' culture. We are also extending the School Games and spending £1 billion on youth sport over the next five years.
"In addition we are working across government on a range of measures to improve PE and school sport as part of the Olympic and Paralympic legacy and will make an announcement in due course."
Shadow schools minister Kevin Brennan said: "This report highlights the importance of PE in schools but as leading sports personalities have warned, under this Government the Olympic legacy is at risk.
"Ministers must restore the requirement that pupils do a minimum of two hours of PE a week - the numbers of pupils doing two hours of sport has collapsed from 90% under Labour to 50% now.
"Much of the progress Labour made in school sports has been lost since the abolition of Labour's School Sports Partnerships."