Officials are to launch a review into home schooling amid fears children could be at risk of radicalisation.
Education Secretary Nicky Morgan has reportedly decided to look into those educated by their parents after the Government previously pledged to clamp down on madrassas teaching anti-Semitic beliefs.
Exact figures for the number of children who are home-schooled are unknown but the Independent on Sunday said there were estimated to be between 20,000 and 50,000 pupils.
The paper quoted a senior Government source as saying: "There has always been the freedom in this country for people to educate their children at home. Many people do it very well. But we need to know where the children are and to be certain they are safe.
"For every parent doing a brilliant job, there may be someone filling their child's mind with poison. We just don't know. We don't have reliable figures."
The Government is considering proposals for parents to have a contact at local councils and to pinpoint how many are home-schooled, the paper said.
A Department for Education spokesman said: "We are determined to tackle radicalisation wherever it occurs.
"We have provided Ofsted with extra inspectors to eradicate extremism in education. We are working with them to address their concerns about home education being exploited, while safeguarding the rights of parents to determine how and where to educate their children."
The review comes after warnings about unregistered religious schools.
Earlier this month, Ofsted chief Sir Michael Wilshaw warned the safety and well-being of hundreds of children could be at risk by their education in unregistered schools illegally operating in several parts of England.
Three unregistered schools shut down by watchdog Ofsted in Birmingham in November were offering a narrow Islamic-based curriculum using anti-Semitic, homophobic and misogynistic material in conditions which represented a fire risk, it said.
Shadow education secretary Lucy Powell said: "This is a worrying weak spot for government.
"It is vital that action is taken to ensure that all children, whether in school or taught at home, are given the knowledge and skills to succeed, not taught a narrow curriculum of hate and bigotry.
"The reality is that there is a dangerous void in the local oversight of our schools system, created by this Government's education policy, and this is allowing children to drop off the radar, where they could be exposed to harm, exploitation or the influence of extremist ideologies.
"We urgently need robust local oversight and accountability of all local schooling, regardless of type, so that communities can work together to improve standards and stop children from ending up in harm's way."