It was previously reported Gary was quitting to concentrate on a new album and tour with Take That in 2014.
Speaking at the premiere of One Direction's film 'This Is Us', Dermot told reporters: "If Simon comes back, the dynamic would be very different. Gary wouldn't be there - he'll be leaving at the end of this year.
"Simon is a great judge so we'd welcome him back, but Gary's been such a great head judge as well."
Simon has been linked with a return to the show, after ITV bosses pleaded with him to make a comeback as part of new contract negotiations to keep the show with the broadcaster.
Gary's departure means there will be at least three vacant seats on the judging panel for the 11th run of the show next year, as Sharon Osbourne has made it clear that her comeback is for one year only, while Louis Walsh has also confirmed he will be departing at the end of the new series.
However, the Irish judge revealed earlier this week he would reconsider his resignation if Simon was to return to the panel.
He told the Radio Times: "I've been doing it ten years. It's an awful long time to be on the show. I don't want to be pushed. I'd prefer to leave than be pushed. To leave on my terms.
"Now if Cowell came back, I might come back. The two of us, we would not stop laughing. If Simon called me up and said... but, oh well. Let's see what happens."