The father of four, who went to school with farmer and presenter Jimmy Doherty, said they were both selected for the special needs stream at their comprehensive, Newport Free Grammar School in Newport, Essex.
Speaking on the set of their new show, Jamie and Jimmy's Friday Night Feast, Oliver said: "While we were at school, I struggled. Imagine a boys' school. Thirty boys in the middle of English, bang bang bang on the door, 'Can we have Jimmy and Jamie for special needs? Just us two out of our class."
He added that the other children in the class would sing 'Special Needs' to the tune of 'Let it Be' when they were called out.
Jamie, who is now worth around £150 million, told The Times he favoured the 'hippy approach' of a school in Los Angeles that his children attended for three months, which taught them using the subjects they love. His daughter Poppy was taught through song-writing.
He added: "I think traditional education has got a lot to answer for. Fifty odd per cent don't leave with five GCSEs, A-C. Not eight or ten per cent. In my mind we're half crap at education."
Speaking about his row with education secretary Michael Gove over academies being allowed to ignore nutritional standards, he said: "I always thought he was very charismatic, it's just I don't like him touching our standards.
"The parents asked us to put them there and there was no logical reason for them to go away; it was a base level that protected our kids.
"They've essentially been put back in slightly different wording. In some respects I'm happy that we're back where we were two years ago, but also upset we've missed two years."
More on Parentdish: Successful people with dyslexia