Bright poor children are around a third less likely to earn a high wage than their less clever, richer classmates, Justine Greening has said.
Disadvantaged youngsters are also more likely to go to a failing school, and take home more than £2,000 a year less as adults, than those born to parents with managerial jobs.
The Education Secretary said much more needs to be done to create a level playing field for all youngsters, to ensure that they can live up to their potential.
Speaking at a Social Mobility Commission conference, Ms Greening said that while progress is being made, the facts on youngsters missing out are still stark.
"Children from high-income backgrounds who show signs of low academic ability at age five, are 35% more likely to become high earners than their poorer peers who show early signs of high ability," she said.
Ms Greening added that a child living in one of England's most disadvantaged areas is 27 times more likely to go an inadequate school than a child in the most advantaged area.
"Graduates from disadvantaged backgrounds who do make it to the top jobs still earn, on average, over £2,200 a year less than their colleagues who happen to have been born to professional or managerial parents, even when they have the same educational attainment, the same role and the same experience."
There are a number of factors that hamper social mobility - ensuring that all youngsters get the same chances in life, Ms Greening said.
These include inconsistent school funding, difficulties in getting good teachers and leaders into schools in disadvantaged areas, and technical education for young people that is not always good enough.
Ms Greening, who grew up in Rotherham, told the conference how a career as a politician was never in her plans as a youngster.
"I'm not someone who ever planned to go into politics," she said.
"Far from it. Actually, some of my earliest memories of politicians are of my father shouting at politicians who turned up on the news, frustrated that he felt they didn't really talk about his life.
"And I always remember my mum telling him that it was pointless shouting at the television because nobody was listening to him. And she was actually right in so many ways.
"But I did steadily grow up to realise that nothing changes in the end without people changing it and that we can make a choice to have things different in the future than they've been in the past and it's got to be a choice that can liberate millions of young people to be able to go for it."
Ms Greening was also asked about the Government's controversial plans to allow grammar schools to expand and new ones to open, and how these selective schools can help to improve social mobility.
"In the end, we're talking about a new model of how grammar schools can work and how selection can work that really does mean that we've got an education system that caters for very different talents and potential of different children," she said.
The minister added: "What we're saying is if we're really going to have things different, we have to be prepared to look at what that's going to take.
"Do we want to make sure, though, that new grammars work effectively, collaboratively to overall lift attainment? Absolutely."