Many PE lessons are failing to improve pupils' fitness, while not enough youngsters are playing competitive sport to a high level, inspectors warned on Thursday.
In a new report, Ofsted raised concerns that many schools are failing to push their sportiest pupils, or help those that are overweight.
It warned that in some PE lessons there is not enough physical strenuous activity, with pupils spending too much time listening to teachers.
Overall, PE lessons are not up to scratch in around a third of primary schools and about a quarter of secondaries, the inspectorate said.
The report, based on inspections of PE in schools over the last four years, concludes that in general the subject is "in good health", with significant investment in the last decade.
But it warns that in more than a quarter of schools, PE teaching did not improve pupils' physical fitness.
Ofsted chief inspector Sir Michael Wilshaw said that PE was part of a child's entitlement to a good education, and that there are some areas for improvement.
"In particular, we found there often wasn't enough physical, strenuous activity in PE lessons. Some teachers talked for too long and pupils were not provided with enough activity to enable them to learn or practise their skills.
"In many of the schools visited, the more able pupils were not challenged sufficiently because teachers' expectations of them were too low."
In a number of primary schools, youngsters were subjected to PE classes with warm-ups that were too short and too easy, followed by "long periods of inactivity" while the teacher talked, the report said.
"Pupils were not challenged to warm up vigorously or build stamina and strength by participating in sustained periods of physical activity," it said.
"They were often prevented from exercising for extended periods because teachers interrupted their learning or took too long to introduce new tasks."
Ofsted also suggested that the most able youngsters were taught the same lessons as their classmates, and given tasks that were "frequently too easy for them."
And it warned that a fifth of primary schools were not making sure pupils could swim at least 25 metres by the time they left.
In secondary schools, sporty pupils were not given enough time to practise and achieve their best, the report said, and only a minority of schools played competitive sport to very high levels.
In general, "Too much teacher-talk and regular interruptions to record information and observe others performing prevented them from remaining physically active throughout lessons," the report said.
Very few schools had set up programmes to meet the needs of youngsters who were overweight or obese, the report added.
Sir Michael said that schools with good PE lessons provided an "ever increasing range of extra-curricular and traditional activities", helping pupils to achieve more.
But he raised concerns about competitive sport in schools.
"Our report found that only a minority of schools play competitive sport to a very high level."
Sir Michael said he is commissioning a fresh report which will compare competitive sport in state schools to that in the private sector.
School sport has become a political hot potato since the Olympics and Paralympics last summer, with calls to ensure that the Games intended legacy "inspire a generation", is fulfilled.
Ofsted's report says that the Department for Education should build on improvements already made to PE and "harness the interest and momentum" generated by the Games to create a new national strategy for school sport.
A new PE national curriculum, published last week, says that between ages five and seven, pupils should be taught to master basic movements such as running, jumping, throwing and catching, take part in team games and perform simple dances.
Between seven and 11, youngsters should play competitive games like football, netball, rounders, cricket and hockey, as well as learn to swim at least 25 metres.
And in secondary schools, students should "use a range of tactics and strategies to overcome opponents in face-to-face competition through team and individual games", develop their technique, take part in outdoor and adventurous activities and compare their performances with previous ones to achieve personal bests.
The new curriculum also says that pupils should take part in competitive sports and activities outside school through community links or sports clubs.
A Government spokeswoman said: "We want all children to be given the opportunities they need to be fit and healthy.
"The draft PE curriculum published last week is designed to put competitive sport back at the heart of school life and end the damaging 'prizes for all' culture. We are also extending the School Games and spending £1 billion on youth sport over the next five years.
"In addition we are working across government on a range of measures to improve PE and school sport as part of the Olympic and Paralympic legacy and will make an announcement in due course."
Shadow schools minister Kevin Brennan said: "This report highlights the importance of PE in schools but as leading sports personalities have warned, under this Government the Olympic legacy is at risk.
"Ministers must restore the requirement that pupils do a minimum of two hours of PE a week - the numbers of pupils doing two hours of sport has collapsed from 90% under Labour to 50% now.
"Much of the progress Labour made in school sports has been lost since the abolition of Labour's School Sports Partnerships."
The Egg And Spoon Race
Stick your egg to your spoon. A variety of adhesives can be used here: glue, blu-tack, chewing gum...
The Relay Race
Grease the other team's relay baton with oil or butter
Sports Day - General Tips
Attempt to trip your opponent up at all times
Sports Day - General Tips
Superglue the soles of your opponent's shoes so they stick to the ground and can't move
Tug Of War
Weigh your pockets down with stones so you weigh more, or just eat A LOT before the competition
The Sack Race
Gluing your feet to the corners of the sack and running
Sports Day - General Tips
Feed your opponent lots of Tic Tacs/Polos/mints so they have to keep running off to the toilet
Sports Day - The Prize
Tell your fellow competitors first prize is a kiss from the elderly head teacher. Who may have a hint of moustache about her lips..