Mothers, however, are 50% more likely to read with their child at 0-11 months than fathers.
The Ipsos MORI poll was commissioned by Book Trust to mark the beginning of National Bookstart Week (8-14 June), an annual celebration to encourage families to read with their children from an early age.
Diana Gerald, Book Trust's chief executive, said: "Now more than ever we need both parents to step up and make time to read with their children because one in five leaves primary school unable to read well."
The charity also found that a quarter more mothers read with their five-year-old compared to fathers.
They said encouraging children to read early helps develop their language skills and provide them the building blocks for reading later in life.
Book Trust said, according to the Office for National Statistics, there are now 229,000 stay-at-home dads, compared to 111,000 in 1993 - yet they are still unaware of the importance of reading.
Gerald explained that reading with your child should be a fun experience, which increases their literacy skills and prepares them for school.
"If a parent reads to their children every day they will be almost 12 months ahead of their age group when they start school," she said.
"Even reading to them three to five times a week gives them a six month head-start over those who are read to less often."
"We believe very strongly that a book is not just a book; it’s a doorway to a better life. And we believe that doorway should be open to everyone."
Writing in her blog for HuffPost UK Parents, Gerald said: "Men read, we know that. So do fathers see reading with children as a female domain? And is this one of the reasons why, by school age, girls tend to read more than boys?"
"It is time to put a stop to this reading habit which has surreptitiously crept in to British culture."
Book Trust is urging not only fathers, but all parents, grandparents and carers to make a promise to read to their children for at least ten minutes every day.
Dad-of-one and founder of The Dad Network, Al Ferguson said: "In essence, reading to your baby is exactly the same as speaking to your baby. Babies need to be surrounded by words and books are the best way to do this. It helps avoid feeling like a wally too!"
Dad blogger, Andrew Watson has also written about the importance of the bedtime story in a HuffPost UK blog: "I may have failed in the nappy-dressing stakes, but at least that was after reading my son a story. It's an inviolable part of the naptime routine, and that's thanks in part to my own father.
"Because as strongly masculine as he was, as bearded and broad and brawny, he regularly read us stories. Not only that, but he was forever reading books himself."
During National Bookstart Week, Book Trust will be giving out 450,000 special editions of Giles Andreae’s beloved Rumble in the Jungle book to families across the country through libraries and children’s centres.