Mr Nasty has gone from his lowly days as tea boy at EMI to creating a TV and music empire which would make Rupert Murdoch quake in his boots.
The X Factor remains the jewel in his crown thanks to its successful debut in the States last year, but there is one woman claiming the glory should be all hers - That's Life! legend Esther.
Yes, that's right, the toothy TV presenter claims she gave Simon his big idea back when he was a young record exec working with the likes of Sinitta.
She asked him to sit on the panel of a talent show segment on her hit show That's Life!, and the rest is history.
Rantzen tells the Scottish Sun: "You know Simon ripped off my idea for X Factor? He was a young record producer when he worked with us on a talent show called Search For A Star on That's Life!
"It was the exact same format as X Factor. We held public auditions, filmed the wannabes - occasionally laughing - and even had the viewers vote for (the) winner."
The 71 year old didn't manage to make a penny from the concept - but she was treated to a slap-up meal from Cowell as a thank you for sparking his million dollar idea.
She adds: "Sadly, unlike Simon and his millions of pounds, I didn't make a single penny. But at least Simon took me for a very expensive lunch at The Ivy as a thank you a few years back and told me, 'The X Factor was all your idea of course.'"