The returning judge - who was axed in 2015 in favour of Nick Grimshaw, only to be reinstated this year - has claimed he loved watching the show flop without him.
Speaking on BBC Radio 2, Louis said: “It's the most talked-about show on TV. I know I wasn't there last year - and it wasn't as good. But I'm back.
“My contract wasn’t picked up last year, I was half expecting it, so it was fine because I had lots of things to do.
“I think being away from the show was a really good thing for me because ratings weren’t good. I knew watching the first show that it wasn’t going to work, so I was going, ‘Yes!’.”
He added: “I called Simon really late that night and told him what I thought, I said, “You’ve messed with it too much, it’s not working’, and he didn’t say anything for a second and then he said, ‘Maybe you’re right’.”
'The X Factor' returns to ITV in the autumn.