Many viewers slammed the new 'cruel' format, which sees successful contestants given one of six seats to go through to the next round, only to face being kicked out if the judges prefer another hopeful over them.
However, Simon has said the changes were 'dramatic' and 'exciting'.
He told The Mirror: "This is some of the most exciting TV we’ve made on one of these shows. It is dramatic, it is controversial, but we all love that.
"People went nuts, half loving it and half hating it.”
On Twitter, some viewers described the new format as a 'bloodsport', comparing it to the movie 'The Hunger Games'.
Singer Lily Allen also raged: “When did become alright to f*** with peoples minds. X factor has got fully mean.”
And comedian Dara O’Briain tweeted: “Wow, X-factor seems to have become more exploitative than ever.”
Axed contestant Lydia Lucy also spoke out against the twist, revealing it is 'one of the hardest things emotionally I have ever had to go through'.
She told The Sun: "It’s so hard to accept because one minute you are through and ecstatic then you have to wait — and eventually get kicked off.
"If Simon Cowell was in front of me now I’d tell him to change the format back to how it was.
"This way is like dangling a carrot in front of someone and then taking it away,” she added.
So who did make it through to judges' houses? Meet the final 24 here...