Cowell, who made his fortune in the music business before starring in the ITV show, said he was "useless" in class but then found a way to achieve success.
He reportedly told US radio station z100: "I didn't work hard when I was at school. I left at 16 and I didn't have any qualifications. I was useless. The secret is to be useless at school and then get lucky."
Cowell, who attended private Dover College and Windsor Technical College, said he suffered many setbacks while trying to forge his career, but it had made him a better businessman.
"You make a ton of mistakes in the beginning and just hope you learn from them," he said.
But Gove accused the media mogul of turning children off education.
"This is an irresponsible and stupid thing to say," he told Mail Online. "Teachers strive every day to ensure children understand the importance of learning, hard work and discipline. Simon Cowell's comments undermine their efforts.
"The truth is that only a tiny fraction of people get lucky. For everybody else, it's all about hard work and that starts with hard work at school. Celebrities like Mr Cowell should encourage education, not rubbish it."