The pair formerly hosted 'X Factor' spin-off series 'The Xtra Factor' together for two years, with Caroline going on to host with comedian Matt Richardson after Olly's departure.
Olly also has a long history with 'X Factor', after first finding fame as a contestant on the show back in 2009, eventually finishing in second place.
Earlier this week, Caroline shared a photo with Olly on her Instagram page, as a 'throwback Thursday' snap to when they "first met".
Back in November, she also uploaded a selfie with him, alongside the caption: "Back working with this lovely chap… will reveal all soon."
Since leaving 'The Xtra Factor' in 2013, Caroline has since gone on to be crowned winner of rival show 'Strictly Come Dancing', though she insists that there were no hard feelings between herself and Simon, despite her having defected to the BBC.
If Caroline and Olly decide they aren't up to the job, here are a few of our suggestions for who could take over from Dermot...
Dermot O'Leary's Potential Replacements
Kate Thornton(01 of12)
Open Image Modal
Kate fronted 'X Factor' way back, when it first arrived on our screens. Could she reprise her role? (credit:JON FURNISS/INVISION/AP)
Jeremy Clarkson(02 of12)
Open Image Modal
Well... he could always give it a go. (credit:ASSOCIATED PRESS)
Fearne Cotton(03 of12)
Open Image Modal
Fearne announced her plans to leave Radio 1 earlier this year, meaning she's definitely free to consider the job. We reckon Fearne would be a fab replacement. (credit:Joel Ryan/Invision/AP)
Zayn Malik(04 of12)
Open Image Modal
We're not sure it fits with his "being a normal 22-year-old" plan, but stranger things have happened... (credit:ASSOCIATED PRESS)
Caroline Flack (05 of12)
Open Image Modal
Get Flacko Backo! The former 'Xtra Factor' presenter went on to win 'Strictly Come Dancing' after being replaced on the 'Xtra Factor' last year. (credit:Joel Ryan/Invision/AP)
Nick Grimshaw(06 of12)
Open Image Modal
Grimmy's no stranger to rubbing shoulders with A-listers, so we reckon he'd make those Sunday night special guests feel right at home. Perhaps his good friend Harry Styles could also co-host a show or two... (credit:Jonathan Short/Invision/AP)
Sinitta (07 of12)
Open Image Modal
It's about time Sinitta and her palm leaves stepped out of the shadows and into the spotlight. (credit:Joel Ryan/Invision/AP)
Steve Brookstein(08 of12)
Open Image Modal
Steve hasn't exactly seen eye-to-eye with Simon Cowell in recent years - perhaps they could patch up their differences and make this happen? (credit:Andy Sheppard via Getty Images)
Eric Cowell(09 of12)
Open Image Modal
YEAH, we know he probably can't talk yet but there are still a few months to go until 'X Factor' returns to our screens. We don't have much experience when it comes to teaching babies to talk, but surely Si can shell out on a tutor or something? (credit:Twitter/Simon Cowell)
Emma Willis (10 of12)
Open Image Modal
The wonderful Emma is a fan favourite thanks to 'The Voice UK' and 'Celebrity Big Brother'. Could she swap the BBC for ITV? (credit:David M. Benett via Getty Images)
Rylan Clark(11 of12)
Open Image Modal
Since starring as a hopeful on the show, Rylan has cut his teeth in the presenting world, hosting a regular segment on 'This Morning' and keeping guests in check on 'Big Brother's Bit On The Side'. (credit:Dave J Hogan via Getty Images)
Khloe Kardashian(12 of12)
Open Image Modal
Khloe received mixed reviews when she presented the US version of the show, but we think she could be a hit here in Blighty. (credit:AJM/AJM)