However, not all hope is lost for them yet, as it appears one or more of them could be returning.
“I wouldn’t rule anything out right now,” he said cryptically.
Louis Walsh is rumoured to be returning
Simon recently teased why there would be a need for a fifth judge on the show, telling The Sun: "There is also something else which we haven't announced yet which will make the logic of having a fifth judge."
Could this be a wildcard category...?
HuffPost UK has contacted an ‘X Factor’ spokesperson for comment.
‘The X Factor’ live shows begin on Saturday (31 October) at 8pm on ITV.
Find out who the final 12 are in our gallery below...