The singer made the admission as he prepares to launch his own rival talent show, ‘Let It Shine’, which will recruit five singers to form a group to star in a stage show about Take That’s music.
The new series, which is set to air on BBC One later this year, is reportedly being closely watched by Simon’s lawyers in case it copies the format of ‘The X Factor’, which might explain why the pair haven’t been in touch.
Gary said: “I haven’t spoken to him since I left X Factor three years ago. My show’s totally separate.”
Gary replaced Simon on ‘The X Factor’ in 2011 and stayed for three series.
But as viewers began to desert the show, Simon blamed Gary for the fall in ratings.
Simon said at the time: “Gary was seen as the head judge - and so am I. I do believe I am the best person at spotting a potential star and making them work well on the show.
“That is what I am good at doing, not being a big-mouth on the show. It is making the show I am on better.”
As well as ‘The X Factor’ and ‘Britain’s Got Talent’, Simon also runs a number of shows overseas, including ‘La Banda’, a South American programme which searches for individuals to make into a band.
Earlier this month, a source told The Mirror: “Simon’s team is keeping a close eye on Gary’s show.
‘Let It Shine’ will be hosted by Graham Norton and Mel Giedroyc, and is believed to follow a similar format to the Beeb’s previous talent shows ‘Over The Rainbow’, ‘I’d Do Anything’, ‘Any Dream Will Do’ and ‘How Do You Solve A Problem Like Maria?’.