Mr Nasty has gone from his lowly days as tea boy at EMI to creating a TV and music empire which would make Rupert Murdoch quake in his boots.
The X Factor remains the jewel in his crown thanks to its successful debut in the States last year, but there is one woman claiming the glory should be all hers - That's Life! legend Esther.
Yes, that's right, the toothy TV presenter claims she gave Simon his big idea back when he was a young record exec working with the likes of Sinitta.
She asked him to sit on the panel of a talent show segment on her hit show That's Life!, and the rest is history.
Rantzen tells the Scottish Sun: "You know Simon ripped off my idea for X Factor? He was a young record producer when he worked with us on a talent show called Search For A Star on That's Life!
"It was the exact same format as X Factor. We held public auditions, filmed the wannabes - occasionally laughing - and even had the viewers vote for (the) winner."
The 71 year old didn't manage to make a penny from the concept - but she was treated to a slap-up meal from Cowell as a thank you for sparking his million dollar idea.
She adds: "Sadly, unlike Simon and his millions of pounds, I didn't make a single penny. But at least Simon took me for a very expensive lunch at The Ivy as a thank you a few years back and told me, 'The X Factor was all your idea of course.'"
Suggested For You
SUBSCRIBE AND FOLLOW
Get top stories and blog posts emailed to me each day. Newsletters may offer personalized content or advertisements.Learn more