With Tulisa apparently ruled out of 'X Factor' this year, speculation is rife about who will sit behind the judges' desk when the tenth series begins filming in a few months.
When pressed about his dream line-up in an exclusive chat with HuffPost UK, Olly backed recent reports linking Mrs O with a return to the show.
"I love Sharon Osbourne - it would be great to have her back," he said.
Robbie has also been linked to a spot on the panel with his Take That bandmate, but Olly insisted he knew nothing about who will be filling the judges' seats this year.
"There's been loads of speculation but we just don't know," he said. "Every year it's the same question and no one ever knows. It's quite exciting, all these reports linking people to the panel."
The singer - who finished second on 'The X Factor' in 2009 - is currently on the road with his own UK arena tour, and is set to join Mr Williams on his European stadium tour later this year.
His hectic schedule has thrown into doubt his return to 'The Xtra Factor', which he co-hosts with Caroline Flack and, when asked about the spin-off show, Olly seemed to confirm rumours of his exit.
"We'll see what happens. Never say never. I've always sat on the fence - it looks unlikely but you never know what might happen," he said.
"I think things will get confirmed in the next month or so as auditions will start in the next few months."
In the meantime, he's already planning the mischief he'll get up to with Robbie while the pair are touring.
"We've already got football kits ready," he said. "It's going to be Robbie's team vs. my team five a-side. It's going to be a great couple of months and it's an honour to be invited."
Olly's current single, 'Army Of Two' is out now.