School sports provision is patchy in some places at the moment, Culture Secretary Jeremy Hunt admitted on Monday.
To continue the British successes of the London Olympics, it is important to ensure best practice is followed around the country, and to have more investment at primary school level, he said.
His comments follow the call by Lord Moynihan, the head of the British Olympic Association (BOA), for a major increase in Government funding to build on the success of Team GB at the London Games.
The BOA chairman accused the current and previous governments of "treading water" in terms of increasing participation.
The peer, a former sports minister under Margaret Thatcher, claimed school sport policy is "bureaucratic" and needs more money to fund a major expansion.
Lord Moynihan said: "There is a need for radical reform and I am calling for more money. There needs to be a total commitment to ensuring a sports participation legacy that has to focus on schools and clubs.
"For seven years successive governments have been treading water.
"We have tens of thousands of kids watching great moments which will live with them for ever. The Government should step up to the mark."
Hunt told BBC Breakfast: "I think at the moment school sport provision is patchy in some places, and we need to do what we can to make sure that the very best examples are spread throughout the whole country, and this is absolutely going to be a focus over the next few months and one of the things that we really want to take away from these Games."
He agreed that investment at primary school level was important, saying: "Primary school is where it all starts, and catching people young is incredibly important."
It was important to keep a sense of perspective, he said.
"When you were showing the medals table just then, we're third in the world. Actually the funding of sport has been one of the great successes. John Major set up the Lottery in the early 1990s, and in the '96 Atlanta Games we won just one gold medal, we've already won 16 in these Games, and we are only halfway through the Games.
"Other countries are now looking at the UK, and looking at our sport funding model, and seeing what they can learn."
Lord Moynihan said children should be given the chance to try minority Olympic sports such as handball, and primary schools help to provide more sporting opportunities. Sport England's focus is on improving participation in the 14-24 age group.
London 2012 chairman Lord Coe said there was "a limited window of opportunity" to seize the day in terms of building on the success of the British team but he believed the Prime Minister, David Cameron, recognised this.
Lord Coe, who sat next to the prime minister in the stadium on Saturday night, said: "He was very seized by the need to build on everything he was seeing in that stadium, whether it was the extraordinary performances of our teams, the ongoing challenges of getting more young people into sport, and also the extraordinary potential for the economic legacy.
"He is very seized by the need to leverage legacy from every nook and cranny of that project. There is a limited window of opportunity."
Labour's former sports minister Richard Caborn accused Lord Moynihan of "rewriting history" by overlooking the contribution to this year's crop of Olympic medals of the state investment in sport.
"We did invest very heavily in sport under the Blair administration," said Mr Caborn. "The success now in the Olympics is a result of the investment we put into UK sports.
"Moynihan shouldn't be rewriting history, he should be reflecting on what we did over the last decade to get us to where we are now. He is not acknowledging the work that was put in over the last decade.
"We put money into school sports and we invested money in elite sport and UK Sport. That has got us where we are."
Sir Menzies Campbell, former Lib Dem Leader and one-time holder of the UK 100m record, said: "I fully support Colin Moynihan's call for greater investment in school sport.
"For far too long in this country we have turned against competitive sport in schools but there is no doubt in my mind that if we are to sustain the wonderful legacy from these Olympic Games we must accept that competition is the very lifeblood of sporting achievement.
"People wonder why Jamaica produces so many outstanding sprinters but we know that the very backbone of sport in that country is a programme of competitive events in schools at every level."