Dannii, who famously had an affair with show boss Simon Cowell, said she is focused on enjoying Kylie's time as a judge over on BBC rival contest 'The Voice'.
The 42-year-old was first rumoured for a return to 'X Factor' after Nicole Scherzinger quit and Sharon Osbourne bowed out.
Simon reportedly wanted to reunite the panel last seen in 2010, which featured Dannii and Cheryl Cole, to reverse the show's declining ratings.
The music mogul was allegedly hoping the tension on the panel will get people talking about the show again, as since the four judges last appeared on the show together, Simon revealed his 2007 fling with Dannii, and he also famously sacked Cheryl from the US version of the show.
Open Image Modal
Dannii Minogue, Louis Walsh, Cheryl Cole and Simon Cowell
"The feeling is that 2010 was the show’s peak because it was the best possible judging line-up - and imagine the tension on that panel now," a source told The Sun.
“Simon knows pairing him with Dannii and Cheryl, while keeping his old mate Louis Walsh to spar with, is a blockbuster combination," the insider added.
Dannii was reportedly looking for new UK projects after 'Britain And Ireland's Next Top Model' was axed, where she served as a judge. But she has now confirmed she plans to stay Down Under.
The original line-up: Sharon Osbourne, Simon Cowell and Louis Walsh
Series 4 (2007)(02 of09)
Open Image Modal
All change: Dannii Minogue joins Simon Cowell and Sharon Osbourne, with Brian Freidman replacing Louis Walsh.Dermot O'Leary also replaces Kate Thornton as the show's host.
Series 4 (2007)(03 of09)
Open Image Modal
The return of the Walsh: Louis makes a comeback after Brian Freidman is axed from the panel after only a few days of auditions.
Series 5-7 (2008-2010)(04 of09)
Open Image Modal
The golden era: Cheryl Cole replaces Sharon Osbourne, joining Simon, Dannii and Louis.
Series 8 (2011)(05 of09)
Open Image Modal
A mass exodus: Gary Barlow, Tulisa and Kelly Rowland join the only original remaining judge, Louis Walsh.
Series 9 (2012)(06 of09)
Open Image Modal
Schermazing: Nicole Sherzinger replaces Kelly Rowland, joining Gary, Tulisa and Louis.
Series 10 (2013)(07 of09)
Open Image Modal
She's BACK! Sharon Osbourne rejoins the panel after five years, replacing Tulisa and joining Louis, Gary and Nicole. (credit:Rex)
Series 11 (2014)(08 of09)
Open Image Modal
REUNITED: Simon and Cheryl return alongside Louis Walsh and new judge Mel B. (credit:Syco/Thames/ITV)
Series 12 (2015)(09 of09)
Open Image Modal
Bye bye Louis: Nick Grimshaw and Rita Ora replace Louis and Mel, as they join Simon and Cheryl on the panel.Meanwhile, Caroline Flack and Olly Murs step into Dermot O'Leary's shoes as host. (credit:Twitter)