Pupils on free school meals are less likely to take part in extra-curricular activities according to a report which highlights "worrying inequalities" in opportunities for children.
Those students are almost twice as likely to see school as "a waste of time", polling by Demos for the Scout Association suggested.
The Learning By Doing report calls on the Government to work to ensure all children, both fee-paying and state school pupils, have the same opportunities for sports, outdoor activities and volunteering outside of school.
Half of all students asked said they would like extra-curricular activities to count towards their GSCEs or A-levels.
Almost two thirds of teachers would like to see non-formal learning being added to the curriculum, according to the report, but the majority of those polled said they did not think there was enough space in the timetable to deliver that learning.
The school census should be used to monitor participation levels in extra activities, and information should be included in the national pupil database, the report recommended.
It also calls for teachers to be trained in learning outside the classroom and character education, and said guidance should be provided on quality provision of non-formal education.
Teachers should be rewarded for efforts to promote non-formal learning in school, the report said, and suggests outside bodies should look at how best to work closely with schools.
Jonathan Birdwell, co-author of the report and head of citizenship and political participation at Demos said the report shows that extra-curricular learning is valued by both pupils and staff.
He said: "The enormous benefits provided by 'non-formal learning' or extra-curricular activities are now well proven, and this report demonstrates just how highly they are valued by teachers and students alike.
"But it also unearths worrying inequalities in the opportunities that are available for children to participate in them. While schools are doing their best to close the gap, we cannot ignore the fact that children on free school meals are conscious they are receiving fewer opportunities, and that teachers feel too constrained by timetabling commitments to deliver them."
He added that school principals are key to the process, by identifying opportunities to deliver a programme of activities.
"Reforms to Ofsted and teacher training will help to embed 'non-formal learning', but ultimately it's up to school heads to forge partnerships with non-formal learning providers that can help to make education more exciting, engaging and better suited to helping all children to succeed both in school and adult life."
Hannah Kentish, UK youth commissioner at the Scout Association, said: "This report shows just how much both teachers and young people alike value the opportunities that non-formal learning can provide. It also highlights that these opportunities are not equally shared across all locations and that much more can be done to embed non-formal learning as a permanent fixture in the school system.
"If we are serious about giving all young people the very best chance to become active citizens, we need to do more of this work. Scouting has so much experience in using adventure-based activities to help young people gain confidence, empathy and resilience and we are ready to help partner with schools to make this important ambition a reality."
A Department for Education spokesman said: "Raising the status of character education is an important part of our plan to deliver real social justice by giving all children, regardless of background, the opportunity to reach their full potential.
"We want all pupils to have access to character-building activities which will help them develop the confidence, resilience and personal skills they need to succeed in adult life and the world of work."