During a disagreement over Gary's contestant Christopher Maloney on Saturday night, Gary told Tulisa she had 'fag ash breath', which visibly shocked the N-Dubz star, with the Take That singer later apologising on 'The Xtra Factor'.
Rylan has now voiced his thoughts on the episode, believing that Gary took his banter too far.
"I think Gary was out of order," he told HuffPost Celeb at Tuesday night's Cosmopolitan Ultimate Women Of The Year Awards.
"He was a little bit too personal, but at the same time it's all about the banter with Barlow. It was probably the red wine!" he added.
After rumoured peace talks between the pair, things still looked tense on-screen during Sunday night's results show, but Rylan insisted Tulisa has got revenge on her mind.
"Tulisa is fine, she's up for the craic. At the end of the day, things do get really heated in that studio and I've had to bite my tongue a couple of times with old GB," he said.
"Tuilsa never lets anything slide. She's the Female Boss. She'll get her own back."
Despite the on-going drama on the judging panel, Rylan is now looking forward to his performance on Saturday.
However, it looks like he isn't going to deliver on his promise to Gary that he would tone down his act.
"We've had a chat with the music team and they've got bigger plans for me," he teased. "I do want to try and showcase a bit of my voice this week.
"I just hope that he (Gary) can one time see what he wants to see from me," he added.
Viewers of last weekend's show also saw Gary's pal Robbie Williams voice his support for Rylan after they bonded during a masterclass from the 'Candy' singer - something which the Essex boy admitted he loved.
"He turned round and said to me, 'look, you remind me of myself when I first started'," he exclaimed. "That's probably the biggest compliment I've had since starting this show."
"Nicole is spending too much time with me - she's turning into me!" he laughed. "I'm taking her to Essex tomorrow for a makeover. We're going to Amy Childs' salon. I may take her into Sugarhut for a bit too."
Now that, we can't WAIT to see.
> IN PICS: THE COSMOPOLITAN ULTIMATE WOMEN OF THE YEAR AWARDS