Heptathleton gold medal winner Jessica Ennis said children needed to learn that competition was positive.
And double gold athlete Dame Kelly Holmes said two hours a week of PE should be made compulsory.
Jessica said: "I started doing athletics when I was nine years old. I had loads of support from my school and PE teachers.
"I think it's really important to have great sport in schools and teach it well and obviously create great role models. That will make a huge, huge difference."
She said it was important to 'teach kids that it is not a bad thing to be competitive' and that she hoped some of the thousands of volunteers and visitors to the Olympics would be inspired to help run local clubs.
Dame Kelly took to Twitter to call for two hours of PE a week – in direct contradiction of David Cameron's earlier declaration that such a target was a box-ticking exercise. She told her 49,000 followers yesterday: "my view for what its worth: PE compulsory min 2 hours, Designated Primary school PE teachers and make stronger links between schools, communities & sport clubs."
Concerns have been raised that Team GB won't be able to repeat its London 2012 success when the Olympics move to Rio de Janeiro in 2018 because of a lack of funds to coach and faciliate the future's sports stars.
But the Prime Minister insisted that simply spending more was not the solution.
Interviewed on LBC Radio, Mr Cameron was asked whether the education department had removed the requirement for two hours of compulsory PE every week in schools.
He said: "Every school has to deliver sport. What the last government did, which is not right, is if you just sit there in Whitehall and set a target but don't actually do anything to help schools to meet it, you are not really solving the problem.
"In fact, by just saying: 'I want you to do this number of hours a week,' some schools think: 'right, as soon as I have met that minimum target, I can tick a box and give up'."
He added later: "The problem has been too many schools not wanting to have competitive sport, some teachers not wanting to join in and play their part.
"So if we want to have a great sporting legacy for our children - and I do - we have got to have an answer that brings the whole of society together to crack this, more competition, more competitiveness, more getting rid of the idea you can't have competitive sports days.
"We need a big cultural change - a cultural change in favour of competitive sports."
Do you think David Cameron is out of time with his comment that teachers don't want to 'play their part'?
How can we encourage more children to enjoy playing sport?
Schools, parents, peer groups?