The singer's decision to leave after this year's series has raised hopes that Simon Cowell will return to the judging panel next year.
Plans are already underway on a new Take That album as well as a mammoth tour in 2014 and 2015 which would mean Gary wouldn't have time to continue in his £1.5million-a-series judging role on the ITV show.
A source told The Sun: “Gary’s a decent bloke and figured it was best to lay his cards on the table as early as possible. He wants to get back to what he does best, recording a new Take That album and touring with his bandmates.
“It looks like there’s no room for X Factor so this is likely to be his last year.
“The producers always feared his day job would get in the way, and they’re delighted they’ve had him for three years.
“I’m sure they’ll present Gary with a plan that could see him do both — they moved auditions in 2010 to allow him to meet his gig commitments — but they know three years on X Factor is likely to be enough for him.”
A spokesman for 'The X Factor' has told HuffPostUK: "This year’s series is not on air yet so it is far too early talk about next year’s series."
And Gary Barlow has told HuffPostUK through his representatives: "Gary has always managed his role on the X Factor with his music in previous years. He hasn’t made any decisions on next year."
Since Simon Cowell left the UK version of 'The X Factor' the show lost 3 million viewers, so show bosses are hoping they can reverse the decline with his return.