The Irish star has appeared on the show every single year, since it first hit our screens, however while chatting on TV, he appeared to suggest that his time on the panel has come to an end.
Speaking on TV3, he explained: ““I think I’m done with ‘X Factor’, honestly. My day job is as a manager, I’ve kind of neglected that. I want to get back to being a manager.
“I’ve done 11 seasons. That’s a long time on television. I think they’re going to change it, go for a few new faces.”
Earlier this week, Simon appeared to confirm that he’ll be back for more on ‘X Factor’, alongside Cheryl Fernandez-Versini who returned to the panel in 2014.
It's also been confirmed that presenter Dermot O'Leary won't be returning to front the show this year, and rumours that Olly Murs and Caroline Flack are to take over have begun circulating.
Ahead of that though, there’s the small matter of ‘Britain’s Got Talent', which returns to our TV screens this evening (Saturday 11 April).
Once again, Si is on the panel, with Amanda Holden, David Walliams and Alesha Dixon.
‘X Factor’ is due to return to ITV in September.