He has now claimed he was exhausted by the show's schedule and the mental pressure of fronting a live TV show.
Speaking to The Sun, he said: "'The X Factor' was amazing but the schedule was hard. Then to add music into that makes it doubly hard.
Olly claimed it hit him hard during the lead up to Christmas, where he had a "week in bed when [I] couldn't move".
Olly and Caroline had a mixed reception from fans after making a series of gaffes during the live shows, the most notable of which saw Olly prematurely announce the departure of contestant Monica Michael.
Caroline was recently forced to speak out on claims she and Olly had fallen out following their doomed presenting stint.
Hitting back on Twitter, she wrote: “I hate being made to make a comment on these things but I feel forced. I’m not bitter about X factor. I have not fallen out with Olly, I wanted to leave, I had the best time.
“I’m happy. Rested. Feel blessed and am looking forward to the rest of the year. Everything evolves and changes . Time to move on xx.”
'The X Factor' returns to ITV this autumn.
Tap the image below to launch the slideshow...