The TV contest lost more than two million viewers on Saturday night, compared to this stage in the competition last year.
The Sun reports that Cowell told producers to “sort it out” in a phone call after last night’s show.
A source told the paper: "Simon is furious. This would never have happened on his watch and he wants the team to know this and sort it out.
"He knows there is great talent on the show, but he thinks the attention to detail has been shocking. The acts are not being showcased at their best.”
Just 11.4million viewers watched Saturday's live ITV show, compared to the 13.7million that watched last year.
Are you an X Factor fan who has switched off from this year’s show? Let us know why below: