And those rumours have been strengthened with the news that bookies have slashed their odds on a Scherzy comeback.
William Hill are backing Nicole to return, with odds of 12/1, while Sharon Osbourne also looks highly likely to be on the 2016 panel at just 2/5.
A source told Now magazine earlier this week: “Simon’s desperate to get 'The X Factor' back in the public favour and he thinks that getting Louis on board again, along with Sharon and Nicole, could be the perfect formula.
"Sharon and Nicole will bring fun to the panel and Simon felt Louis was the missing link last year. They’re good friends and have been talking about how to make the show great again."
However, Rita Ora still hasn't announced whether she intends to return to the show this year, and previous reports claimed that Simon was desperate to resign her.
Nicole was a hit with viewers when she first appeared as a judge from 2012 to 2013, and won her first series with James Arthur.
She was awarded the role after impressing with two-guest judging stints - the first in 2010, saw her help Simon and Louis put One Direction together.
The former Pussycat Doll is also clear to return now that Cheryl has officially left, as they pair have a long-running feud.
Cheryl previously branded her "embarrassing" in her autobiography, recalling their first meeting.
"She started singing 'Promise This' to me, and I swear to God the woman sang the whole song, to my face. It was just so awkward, and every time I stood next to her she started singing it again."
Things got even more awkward when Nicole replaced Cheryl on the US version of 'The X Factor', after the former Girls Aloud singer was fired after just two weeks in the job.
Also on HuffPost
Suggested For You
SUBSCRIBE AND FOLLOW
Get top stories and blog posts emailed to me each day. Newsletters may offer personalized content or advertisements.Learn more