The Take That star joined the show in 2011, but this season has seen him involved in a string of on and off air spats - and, as a result, Gary will not be returning for the 2013 season, according to The Mirror.
A source said: "It's time for a shake-up. While Gary has never committed to another series, he was willing to listen to offers if this one went well.
"However, he was given the least desirable category - the Overs - and three of his four acts have already been eliminated. He feels his integrity has been compromised and the show has lost its credibility. By mutual consent, he will not be returning next year."
Meanwhile, The Sun reports fellow judge Louis Walsh is eyeing a return for Simon on the next series in a bid to boost dwindling ratings.
The paper quotes the Irishman as saying, "If Simon did come back, he'd have the number one show. That's the thing about Simon - he always wants to be number one.
"We need him in the building, he's always fun, he keeps everybody on their toes. I wish he'd come back."
However, a spokesperson for the show has downplayed the rumours that Simon is said to be looking to resume his role as head judge.
The rep tells The Mirror: "We are currently focused on this series. No one is even thinking or talking about next year and who may or may not be on the panel."
But the rumours seemed to be the last thing on Gary's mind on Monday night as he was honoured with the prestigious Music Industry Trusts Award in recognition of his services to British music and charity.
Sir Elton John handed him the prize, and the star gushed, "I've been doing this for 22 years now, and the one thing that's more apparent to me than ever before is how much I love performing.
"To be able to do it, I feel really lucky, I never want to turn my back on it. It's so special when you get something like this, because it's recognised within the industry. I'm a lover of this industry, I'd do anything to protect it and when you're given something from that industry, I'm honoured."
He also teamed up with his Take That bandmates to perform a string of their hits, prompting Robbie Williams to declare on his Twitter.com page, "There is nothing better than having the band back together for a moment."
> IN PICS: X FACTOR'S BIGGEST LOSERS