But their rivalry is bound to step up a gear after Gary hears that Simon reacted badly to the Take That singer being awarded an OBE earlier this year.
In a new chapter of Sweet Revenge: The Intimate LIfe Of Simon Cowell, biographer Tom Bower claims that Simon believed Gary "chased" the award by organising the Diamond Jubilee Concert.
"The announcement that Barlow would receive an OBE for arranging the event added to his irritation. 'He chased that madly to get three initials after his name'," Bower wrote.
"The ambitions of Barlow, he believed, were comical. Ever since Barlow had clawed his way back after a disastrous attempt to become a solo star, Cowell was niggled by his conviction that Barlow was bidding to establish a rival entertainment empire and, like Paul McCartney, secure a knighthood.
"He enjoyed snubbing the man who foolishly asserted that he was the top judge, not Louis Walsh, by texting his criticism about Barlow to others."
Is that a waft of jealousy we smell, Mr Cowell?
The new chapter has been published for the paperback version of the biography released earlier this year, which exposed his affair with Dannii Minogue in 2007.
Meanwhile, Tom Bower has told Heat magazine that Simon will definitely be returning to the UK series of X Factor at some point this year.
“There’s no doubt he thinks he’ll come back for the final. The role that he’ll play will become clearer in a few weeks," he teased.
> IN PICS: X FACTOR 2012