The singer also joked that his Take That band mate wouldn't be as good as him as a judge either.
"I don’t know about [Robbie]. I know how much time and patience you need for this show so I don’t know.
"I know he’s still flying as an artist and if I was to ask him I’m pretty sure he would say it’s not the right time."
And Gary also had some career advice for Sam Bailey - who is hot favourite to be crowned the winner of this year's series.
"If they do a covers album with her then they are committing suicide," he warned. "They need to get a team of young writers like Celine [Dion] has just done.
"She’s made a contemporary record of young writers, is relevant in the charts today and on radio. That’s what they should do with Sam.
Earlier this week, we caught up with the former prison officer ahead of the 'X Factor' final this weekend to find out how she's coping with the pressure of being the favourite to win...
MORE 'X FACTOR'