The 61-year-old has been a judge on all 10 series of the music-based reality show, and even stood in for Simon Cowell on the American version of the show while he was suffering from bronchitis.
However, after it was announced earlier this week that Cheryl Cole would be returning to the show, it was reported that she wanted another female to complete the panel along with her and Simon Cowell - meaning there would be no room for Louis.
He has since hit back at claims that he won't be returning for the 11th series, telling The Sun: "I'm not worried."
A friend of his told the paper: "Louis is tight with Simon and isn’t bothered about what Cheryl has been saying."
It's also been revealed that Louis already has his own ideas about who should fill the fourth judging spot, as the friend added: "He will be back as far as he’s concerned — and he wants Sharon Osbourne at his side.”
Pop singer Rita Ora has already turned the job down as she reportedly wants her music career to be her priority.