It was announced on Friday afternoon that after eight years with the show, Dermot would not be returning to host the ITV singing competition, with reports claiming that he'd been axed by Simon Cowell.
The Sun have now named Caroline and Olly as potential replacements, with a source telling the newspaper: "Simon has secretly met with Olly and Caroline. They’re all fully on board. Olly is very much the new Dermot. It’s a huge coup because of his music commitments."
Caroline Flack and Olly Murs
The pair formerly hosted 'X Factor' spin-off series 'The Xtra Factor' together for two years, with Caroline going on to host with comedian Matt Richardson after Olly's departure.
Olly also has a long history with 'X Factor', after first finding fame as a contestant on the show back in 2009, eventually finishing in second place.
Earlier this week, Caroline shared a photo with Olly on her Instagram page, as a 'throwback Thursday' snap to when they "first met".
Back in November, she also uploaded a selfie with him, alongside the caption: "Back working with this lovely chap… will reveal all soon."
Since leaving 'The Xtra Factor' in 2013, Caroline has since gone on to be crowned winner of rival show 'Strictly Come Dancing', though she insists that there were no hard feelings between herself and Simon, despite her having defected to the BBC.
Last September, she told HuffPost UK: "We're on good terms."
Meanwhile, the news of Dermot's shock departure from 'X Factor' comes just days after he was named one of the hot favourites to replace Jeremy Clarkson on 'Top Gear'.
If Caroline and Olly decide they aren't up to the job, here are a few of our suggestions for who could take over from Dermot...