The minister backed parents who have called for their children to delay going to school, telling councils and head teachers to do 'what's in the best interests of the child'.
Campaigners says some local authorities make it difficult for families who want to put off joining reception classes.
The Department for Education issued guidance to schools and councils in July 2013 and last December making it clear that there is no rule stating that a child has to be in the right school cohort for their age group.
Summer-born children can wait until the September after their fifth birthday if their parents feel they need more time to develop.
And it suggests delaying the start of school by 12 months should be 'no barrier' to the child then starting in reception class.
The rule was introduced after research showed they were more likely to struggle at school, do worse in exams and have less chance of getting into university.
But some councils are still sending children straight into Year 1 if their parents hold them back unless they can prove there were 'exceptional' circumstances, the BBC reported.
Mr Gibb told MPs that there had to be 'flexibility' from schools and councils when agreeing the start date for children who turn four just weeks before the start of the school year.
Giving evidence to the Education Select Committee, he said: "The key thing is flexibility, and I think that admissions authorities and head teachers do need to do what's in the best interests of the child.
"If a child is very immature for their age and they are born in those summer months, and it's the very strongly held view of the parents that the child should start school in the following September, which is when the law says they should start, I think admissions authorities should be flexible and should do what's in the best interests of the child."
He stressed that councils must include parents in discussions about when their children start school.
Michelle Melson, of the Summer Born Campaign, told the committee: "We do have a compulsory school age in England, so children should be able to start at compulsory school age."
She added that summer-born babies who do start school after they are five are starting at compulsory starting age and should not miss out on reception by being placed straight into Year 1.
Ms Melson said: "Parents are being forced to enrol their child in school early, prior to compulsory school age, in order to have a reasonable chance of getting a place in their preferred schools."
More on Parentdish: Summer-born children can start school a year later