The show boss has long hinted he might come back to spice up the show and now he's told The Metro he will return "at some point".
"I still miss that, I mean that show is my baby and that last year we did which was the One Direction year was the most fun I’ve had – it was fantastic," he added.
The soon-to-be father also revealed that he has signed a new 3-year-deal with ITV to keep the show on air - there go all the rumours that the singing contest is coming to an end.
And asked whether Cheryl Cole - his old judging pal and 'X Factor' favourite - will return to the show with him, Simon said: "She was a great judge. I mean, apart from being cute, she’s a great judge and she knows what she’s talking about and, oh my God, she wears her heart on her sleeve!"
He also hinted that Rita Ora could earn herself a place on the show after meeting with her recently.
This weekend on 'X Factor', the show will celebrate its 10th anniversary with the remaining finalists performing hits by former winners.
Former alumnus Olly Murs will also take to the stage to perform. And if we've learned anything from the past 10 years of the show, he'll probably be in a pair of tight trousers.
MORE 'X FACTOR'