Frankie - who was kicked off the show in 2011 for taking cocaine - also said if he ever had the chance to meet Cowell he'd 'spit in his eye'.
When asked if he'd ever met the music mogul, Frankie told Star magazine: "Nah. I'd like to meet him now so I could spit in his eye.
"No, we only had a Skype conversation. He made the background look proper fancy, cocky bastard, sitting in LA.
"You don't need to put the laptop there, do you, Si? Put it against the f**king wall!"
And there's no love lost between Frankie and Louis Walsh either. The 'X Factor' judge made no secret of the fact he wasn't a fan of Frankie while he was on the show.
"I don't know why he didn't f**king like me. He's a prick," Frankie told the mag.
But there were two people, at least, who Frankie was a little more complimentary about.
"Put Robbie [Williams] on [The X Factor]. That would be f**king brilliant. He spent so much time with us when we weren't filming," he said.
And Frankie's also a fan of new judge Nicole Scherzinger, saying: "Yeah, I thought she was f**king good. I loved Kelly Rowland but Nicole pissed all over her to be fair, and she pissed all over Tulisa, too."
Meanwhile, Dannii Minogue is reportedly set to become a judge on the Australian version of 'X Factor'.
The former UK judge is said to be close to signing on the dotted line in a six-figure deal which would see her joining Ronan Keating on the panel.