The former Pussycat Doll's decision to leave comes just days after Simon Cowell confirmed he will be making a comeback on the panel of the new series.
A friend of the US star told The Sun: “It breaks her heart, but Nicole is sad not to be able to go back to the show this year. She has agonised over this decision for months.
“She put her music career on hold last year but now she has to focus on her music.”
Simon is hoping to win back viewers to the ITV singing contest after the last few series failed to live up to expectations.
Key to reviving the show is Simon's return but show bosses are also desperate to sign former Girls Aloud star, Cheryl who left in 2010.
She has reportedly been given a six week deadline to make a decision after Simon confirmed she is in talks with ITV.
A source told The Mirror: “She is in a good place, the best she’s ever been.
“She is focused on her music career and doesn’t know whether she wants to throw herself back into the spotlight to this degree again.”
More than 19million viewers tuned in to see Matt Cardle win the show then compared with just 11.5million last year when Sam Bailey triumphed.
Louis Walsh is also returning for the 11th series, with Rita Ora rumoured to be completing the judging line-up.
Let's remind ourselves of what a 'schamazing' 'X Factor' judge Nicole Scherzinger has been in our round-up of her most memorable moments below...