The former judge told The Sun: “I don’t know who is on the panel but it won’t be me.”
Dannii, who famously had an affair with show boss Simon Cowell, said she is focused on enjoying Kylie's time as a judge over on BBC rival contest 'The Voice'.
The 42-year-old was first rumoured for a return to 'X Factor' after Nicole Scherzinger quit and Sharon Osbourne bowed out.
Simon reportedly wanted to reunite the panel last seen in 2010, which featured Dannii and Cheryl Cole, to reverse the show's declining ratings.
The music mogul was allegedly hoping the tension on the panel will get people talking about the show again, as since the four judges last appeared on the show together, Simon revealed his 2007 fling with Dannii, and he also famously sacked Cheryl from the US version of the show.
Dannii Minogue, Louis Walsh, Cheryl Cole and Simon Cowell
"The feeling is that 2010 was the show’s peak because it was the best possible judging line-up - and imagine the tension on that panel now," a source told The Sun.
“Simon knows pairing him with Dannii and Cheryl, while keeping his old mate Louis Walsh to spar with, is a blockbuster combination," the insider added.
Dannii was reportedly looking for new UK projects after 'Britain And Ireland's Next Top Model' was axed, where she served as a judge. But she has now confirmed she plans to stay Down Under.