Saturday's show garnered an average of just 7.6 million viewers - the lowest figure of the series and the worst for this stage of the competition since the first season in 2004.
To add insult to injury, BBC rival 'Strictly Come Dancing' waltzed off with a peak audience of 10.8 million, and also boasted the biggest lead over 'The X Factor' in six years.
Now, The Mirror claims music mogul Simon - who now serves as a judge on the US version of the show - is to form an urgent plan to "stop the rot".
"Simon always wants to be number one, but X Factor is not just second, it's a long way behind. It's never been this bad and Simon and those he has left in charge will be hurting.
"The only way of getting things back might be for Simon to return and save the day, but he has got his hands full in America."
> IN PICS: X FACTOR'S BIGGEST LOSERS