The music mogul is hoping the tension on the panel will get people talking about the show again, as since the four judges last appeared on the show together, Simon revealed his 2007 fling with Dannii, and he also famously sacked Cheryl from the US version of the show.
Dannii Minogue, Louis Walsh, Cheryl Cole and Simon Cowell
"The feeling is that 2010 was the show’s peak because it was the best possible judging line-up - and imagine the tension on that panel now," a source told The Sun.
“Simon knows pairing him with Dannii and Cheryl, while keeping his old mate Louis Walsh to spar with, is a blockbuster combination.
“Dannii and Cheryl both have a lot of reasons to be furious with him, for personal and professional reasons, but Simon’s a businessman. He knows it will make great viewing,” the insider added.
The current 'X Factor' panel
Dannii was first linked to a return to ITV last month, and is now looking for new UK projects after 'Britain And Ireland's Next Top Model' was axed, where she served as a judge.
Cheryl is now friends with Simon again following her sacking, and a source also told The Mirror on Saturday that she is being courted by her old boss.
The panel last seen together in 2010
"Simon wants Cheryl to join him on the judging panel again. He regards the two of them as the dream team," they said.
"Crucially, she also attracts headlines and will get people talking about 'The X Factor' again. It's still very early days, so nothing is set in stone.
"At the moment talks are ongoing while Simon focuses on the remainder of the year."
However, a spokesperson for Cheryl denied she had even been approached about a comeback when contacted by Heat magazine.