Grimmy was unable to attend the second day of the final challenge, and appeared via Skype as Simon, Cheryl Fernandez-Versini and Rita Ora tried to whittle down the hopefuls.
Open Image Modal
Nick Grimshaw
However, Simon soon grew tired of his interruptions, and shut down the laptop to cut off the Skype call.
It was reported last week that Louis Walsh would be returning in time for the live shows, acting as a fifth judge, and Simon remained coy when asked about the possibility.
"We always had an idea about maybe doing that with different people on the live shows, so we'll see. It's going to be an interesting couple of weeks," he said.
Louis admitted he was keen to reclaim his set on the panel during an appearance on ‘Loose Women’ last week, saying: “There is a possibility I might end up on the show... But I don’t know for definite. I’d like to. I’m available.”
‘The X Factor’ continues on Sunday (4 October) at 7pm on ITV.
'X Factor' Judges Ranked Worst To Best
10. Gary Barlow(01 of10)
Open Image Modal
Gary Barlow spent his entire time on the 'X Factor' panel banging on about how important it was to showcase 'authentic' talent, then sending Frankie Cocozza out in a vest top to perform the Black Eyed Peas song 'I Gotta Feeling'.No ta, Gary. (credit:Ian West/PA Archive)
9. Kelly Rowland(02 of10)
Open Image Modal
We know there's a lot of love out there for Kelly, but she didn't half make every situation about herself, which is why we've lumped her so far down this countdown. She also once literally bowed down at Gary Barlow's feet, which is a big no-no in our book.Mind you, she was involved in one of our fave ever 'X Factor' moments when she quite literally phoned in sick, putting on a special 'woe is me I've got a sore throat' voice to deliver her vote live from LA. Embarrassing and fabulous in equal measure. (credit:Gregg DeGuire via Getty Images)
8. Tulisa(03 of10)
Open Image Modal
Now we know there's a lot of people out there who aren't big fans of the 'Young' singer, but for the two years she was on 'The X Factor' she did have quite a few brilliant moments, particularly when she held her own against a contestant who branded her a low-rent Cheryl Cole - not to mention maintaining her dignity when Gary 'The Lad' Barlow slagged off her "fag ash" breath.She'd probably be higher up the list if she didn't insist on doing that stupid thing with her arm... (credit:Suzan/Suzan)
7. Mel B(04 of10)
Open Image Modal
Now we like Mel a whole lot, and we have high hopes for her time on 'The X Factor', but frankly she's only been in a couple of episodes, so it wouldn't really be fair to put her too far up the list, would it? (credit:Danny Martindale via Getty Images)
6. Louis Walsh(05 of10)
Open Image Modal
Oh, Louis. He's like a piece of the 'X Factor' furniture at this stage. Without lovely Louis putting his foot in his mouth, winding everybody up, giving Irish contestants fast passes to the final and giggling away while people deliver their sob stories, what would 'The X Factor' even be? (credit:Alex Huckle via Getty Images)
5. Dannii Minogue(06 of10)
Open Image Modal
After years of living in her sister's shadow (Kylie, that is, perhaps you've heard of her...) Dannii really came into her own on 'The X Factor'. The Australian star brought a touch of glamour to the panel with her arrival, and we looked forward to her hairstyles, her dresses and - of course - her frosty relationship with Sharon Osbourne every week.Dannii mentored not one but two winners during her time on the show - but don't get too excited, they were Leon Jackson and Matt Cardle. (credit:Mark Cuthbert via Getty Images)
Let's be honest, Cheryl's arrival on the 'X Factor' scene in series five really helped catapult the show to iconic status. The whole country fell in love with her Bambi eyes, her Geordie one-liners (who wasn't saying "you are right up my street" for weeks after Cher Lloyd's audition) and the way she could put Simon Cowell in his place. Or at least she could until she threw on those purple flares and he sent her packing. Still, they've made up now and they're both back on the panel, and it feels like nothing's really changed (besides her fancy new surname, that is). (credit:Alex Huckle via Getty Images)
3. Sharon Osbourne(08 of10)
Open Image Modal
Whether she's boastfully laughing in a contestant's face, chucking a glass of water in Louis Walsh's face or just telling everyone how faaaabulous she thinks they are, Sharon Osbourne really came (credit:Ian West/PA Archive)
2. Nicole Scherzinger(09 of10)
Open Image Modal
One word for Nicole Scherzinger dishing out sausage rolls in a Greggs tabard? Schermazing. (credit:Doug Peters/Doug Peters)
1. Simon Cowell(10 of10)
Open Image Modal
Really, who else could we have had at the top?On paper, everything about Simon Cowell shouldn't work - a man in his 50s, shirt unbuttoned to the naval, with two dogs named Squiddly and Diddly deciding who can and can't sing, yet for some reason we can't help but be charmed by him, despite his status as TV's original Mr. Nasty.He can crush your dreams, insult your clothes and send you up the river all in the name of light-hearted entertainment, but one flash of those pearly whites and he's immediately forgiven... (credit:Tim P. Whitby via Getty Images)