The pair have both fallen ill ahead of the start of filming in Leicester on Friday (10 June).
The Sun has reported it is "touch and go" as to whether they will both make the first auditions, after Dermot was struck down with a severe bout of flu, and Simon struggles to recover from the flu.
Dermot - who was reinstated as 'X Factor' host earlier this year, after being replaced by Caroline Flack and Olly Murs in 2015 - even pulled out of an appearance at the Glamour Awards earlier this week due to his illness, leaving wife Dee Kopang-O'Leary to attend alone.
Simon was also absent from his own charity ball for Together For Short Lives last week, leaving girlfriend Lauren Silverman to entertain guests.
A source told the paper: “Dermot is a regular guest at the Glamour Awards so his absence proved how ill he is.
However, an 'X Factor' source told HuffPost UK: "Both Simon and Dermot have been unwell but we don’t expect any issues on the first day of auditions."
'X Factor' auditions were postponed last year, following the death of Simon's mother, Julie.
'The X Factor' returns to ITV in the autumn.