Nicole's Formula 1 boyfriend, Lewis Hamilton, has allegedly asked her not to return to show because he fears they will be apart for too long, the Sunday People claimed.
When 'The X Factor' returns in autumn, Lewis will be touring the world with Mercedes AMG racing team and Scherzinger will be required to travel the UK in search of new singing talent.
An insider told the newspaper: "Lewis has said to Nicole this is going to be an awful situation and perhaps she shouldn't do it.
"If he thought she had her heart set on returning he wouldn't say this, but he knows she isn't desperate to go back and thinks she would be happier if they were together.
"Nicole is also worried she won't see much of Lewis, so it is a big factor that will deeply affect her decision."
However, Scherzinger's representatives have denied these claims, the Daily Mail reports.
A £1m offer is said to be on the table to secure Nicole for a second series, but with her album due out in November, she will need to be available for promo and is worried she will not be able to fully commit to 'X Factor'.
A pal of the star told The Sun she had said: “I’m so torn right now. My heart wants me to come back and do the show as I had such a great time last year, but my album is coming out and I also need to focus on that.
“I don’t think I can make it work. By trying to juggle the two, one or both will suffer.”
She also cited fans reacting badly to former judge Kelly Rowland when she jetted back and forth to LA and missed a live show due to illness her preventing from flying back in 2011, with Nicole adding that she didn't want that to happen to her.
However, those close to the star are urging her to sign up for the show, according to the paper.
Meanwhile, Simon Cowell has promised to revamp "every aspect of the show" following last year's falling viewing figures.
Already having revealed that the original judging room auditions were returning, he told the Radio Times: "The format is ten years old and any format that has lasted that long, you have to change the format because the inevitable result is if you don't [the ratings] will go down.
"Every aspect of the show [will be changed] in some particular way. It will look like a slightly different show to one you have seen before.
"My job is to try and make the show better than it was two or three years ago."
The original line-up: Sharon Osbourne, Simon Cowell and Louis Walsh
All change: Dannii Minogue joins Simon Cowell and Sharon Osbourne, with Brian Freidman replacing Louis Walsh. Dermot O'Leary also replaces Kate Thornton as the show's host.
The return of the Walsh: Louis makes a comeback after Brian Freidman is axed from the panel after only a few days of auditions.
The golden era: Cheryl Cole replaces Sharon Osbourne, joining Simon, Dannii and Louis.
A mass exodus: Gary Barlow, Tulisa and Kelly Rowland join the only original remaining judge, Louis Walsh.
Schermazing: Nicole Sherzinger replaces Kelly Rowland, joining Gary, Tulisa and Louis.
She's BACK! Sharon Osbourne rejoins the panel after five years, replacing Tulisa and joining Louis, Gary and Nicole.
REUNITED: Simon and Cheryl return alongside Louis Walsh and new judge Mel B.
Bye bye Louis: Nick Grimshaw and Rita Ora replace Louis and Mel, as they join Simon and Cheryl on the panel. Meanwhile, Caroline Flack and Olly Murs step into Dermot O'Leary's shoes as host.