The pair - dubbed Carolly by fans - announced their departure as co-hosts of the ITV singing competition earlier this year, amid reports they had been axed by Simon Cowell after just one series.
Now, Caroline has hit back at rumours she and Olly are no longer speaking, after The Sun claimed she was angered by the fact he had learnt of their fate before she did, but failed to give her a heads up.
Writing on Twitter, she said: "I hate being made to make a comment on these things but I feel forced. I'm not bitter about X factor. I have not fallen out with Olly, I wanted to leave, I had the best time.
"I'm happy. Rested. Feel blessed and am looking forward to the rest of the year. Everything evolves and changes . Time to move on xx."
“His infamous mistake on the live results show last year can’t have helped either, she feels. Of course, Olly has his hugely successful pop career and didn’t really need 'The X Factor', but that was Caroline’s dream job."
Her spokesperson also told HuffPost UK: "The story is absolute rubbish. Caroline and Olly remain firm friends."
Olly's reps declined to comment.
Caroline and Olly, who have always given so many friendship goals, came in for harsh criticism during their one-year stint as 'X Factor' hosts, having previously proved popular while co-presenting 'Xtra Factor' together from 2011 to 2012.
One, now infamous, mistake saw Olly announce the departure of Monica Michael from the competition before the judges had made their decision during Deadlock.
However, their on-screen chemistry led to constant speculation that they were a couple, but the pair constantly denied it, insisting they were just the best of friends.
Back in December, Caroline said that she and Olly were closer than ever after fronting the ITV show, telling 'Lorraine': "We've always got on really well, but for some reason I didn't think it was possible to get any closer.
"This series we've become so close, we've kind of had to stick by each other and we are just friends, but there's this innate understanding between us which I've never had with anyone."
Meanwhile, Dermot O'Leary has succeeded them both as the new 'X Factor' host, having previously fronted the show for eight series until 2015.