The presenter - who used to date One Direction's Harry Styles - has confirmed she won't be returning to the show for the next series.
The news comes just a day after former Spice Girl Mel B was announced as the fourth judge on 'The X Factor' panel.
Confirming her departure from 'The Xtra Factor', Caroline said: "I have a lil announcement. After three brilliant years of hosting Xtra Factor its time to pass the baton. It's been incredible and I'll never forget it.
"How can I when I am married to Louis Walsh? It's all he ever talks about at home. I wish the new team all the best and look forward to watching the new series."
The 22-year-old comedian replaced Olly Murs on the show last year.
New girl, Sarah Jane Crawford, 29, will now present the programme on her own.
Speaking of her new role, Sarah-Jane said: “I’ve been a huge fan of The X Factor since day 1 and have watched every series religiously, so to be the new host of The Xtra Factor in its most exciting year EVER, is what dreams are made of.
"I'll be asking the judges the questions that people at home are screaming at their TVs - and you'd better believe I'm going in with a bang! I just can’t wait to get on the road now and meet the auditionees!"
Meanwhile, an X Factor spokesperson said: "After three amazing years Caroline has chosen to move on from the show.
"As one of the longest serving Xtra Factor hosts ever, she did a brilliant job at getting all the goss and behind the scenes news and chats. We wish her all the best as she heads off on to new adventures.
“Matt had a great partnership with Caroline and brought a comedy edge to his interviews. We’ll miss the fun factor that he brought to the show and which we’re sure he will take to his next project.”
Angela Jain, ITV Director of Digital Channels added: "Caroline was a fantastic and enthusiastic host of the show and we will miss her hugely and wish her the best of luck and a big thank you to Matt for his hard work and humour.
"We are in discussion with both about other projects and look forward to working with them again in the future."
'The X Factor' and 'Xtra factor' are due back on our screens in September.