The music mogul is hoping the tension on the panel will get people talking about the show again, as since the four judges last appeared on the show together, Simon revealed his 2007 fling with Dannii, and he also famously sacked Cheryl from the US version of the show.
Dannii Minogue, Louis Walsh, Cheryl Cole and Simon Cowell
"The feeling is that 2010 was the show’s peak because it was the best possible judging line-up - and imagine the tension on that panel now," a source told The Sun.
“Simon knows pairing him with Dannii and Cheryl, while keeping his old mate Louis Walsh to spar with, is a blockbuster combination.
“Dannii and Cheryl both have a lot of reasons to be furious with him, for personal and professional reasons, but Simon’s a businessman. He knows it will make great viewing,” the insider added.
The current 'X Factor' panel
Dannii was first linked to a return to ITV last month, and is now looking for new UK projects after 'Britain And Ireland's Next Top Model' was axed, where she served as a judge.
Cheryl is now friends with Simon again following her sacking, and a source also told The Mirror on Saturday that she is being courted by her old boss.
The panel last seen together in 2010
"Simon wants Cheryl to join him on the judging panel again. He regards the two of them as the dream team," they said.
"Crucially, she also attracts headlines and will get people talking about 'The X Factor' again. It's still very early days, so nothing is set in stone.
"At the moment talks are ongoing while Simon focuses on the remainder of the year."
However, a spokesperson for Cheryl denied she had even been approached about a comeback when contacted by Heat magazine.
The original line-up: Sharon Osbourne, Simon Cowell and Louis Walsh
All change: Dannii Minogue joins Simon Cowell and Sharon Osbourne, with Brian Freidman replacing Louis Walsh.
The return of the Walsh: Louis makes a comeback after Brian Freidman is axed from the panel after only a few days of auditions.
The golden era: Cheryl Cole replaces Sharon Osbourne, joining Simon, Dannii and Louis.
A mass exodus: Gary Barlow, Tulisa and Kelly Rowland join the only original remaining judge, Louis Walsh.
Schermazing: Nicole Sherzinger replaces Kelly Rowland, joining Gary, Tulisa and Louis.
She's BACK! Sharon Osbourne rejoins the panel after five years, replacing Tulisa and joining Louis, Gary and Nicole.
REUNITED: Simon and Cheryl return alongside Louis Walsh and new judge Mel B.
Bye bye Louis: Nick Grimshaw and Rita Ora replace Louis and Mel, as they join Simon and Cheryl on the panel.