The media mogul released a statement after it was confirmed that judges Abdul and Nicole Scherzinger and host Steve Jones would not be returning for season two of the talent show, thanking them for their work and suggesting that he would be teaming up with Paula again.
He said: "I want to say a massive thank you to Paula, Nicole and Steve for being part of the X Factor last year."
He continued: "You do develop friendships with the people you work with and Paula, in particular, is a very close friend and I expect to be working with her on another project in the near future."
Cowell said he was sure all three would "have massive success in what they do next".
Abdul confirmed her exit earlier this week. She said in a statement she has learned that "business decisions often... override personal considerations" and that she and Simon remain friends.
Former T4 presenter Jones called his departure "a shame" on Twitter and added that he could not complain "as I've had a great time". He gave no reason why he was not returning to the singing contest.
The departure of the three stars is thought to have been part of a shake-up on the series, which failed to achieve the ratings that had been expected.
Cowell and record producer LA Reid will now be the only judges returning for the second season. Replacements for Paula, Nicole and Steve have not been announced.