According to The Sun, Olly will join Gary Barlow, as he chooses which of his Groups to take to the live shows, Nicole Scherzinger has chosen Mary J Blige to help her whittle down the Girls category in Antigua, while in the south of France, Louis Walsh will have two helpers in the shape of Sharon Osbourne's daughter Kelly, and Westlife's Shane Filan as he picks his Boys.
It is thought Sharon is still deciding on who will join her in LA to chose which of the Overs will make it to the live shows.
However, the news has not yet been confirmed by the show, with an 'X Factor' spokesperson telling The Sun: “We have yet to confirm details of the judges’ houses. The first people we will tell will be the contestants.”
Last month, Gary told HuffPost Entertainment that he would like former panellist Cheryl Cole to join him again at Judges' Houses, having previously asked her the year before.
Asked if he had approached the former Girls Aloud singer to help him pick his finalists again, Gary told us: "I don't know yet, but who knows? I love Cheryl, love Cheryl.
"I always ask her to do things, we did the Jubilee together, so I would have her back in a heartbeat."
Earlier this week, it was reported that ITV bosses are begging Simon Cowell to return to the show next year, as they begin negotiations to extend the show's contract with the broadcaster.