The long-standing judge announced he would be departing the show earlier this month, claiming he 'wanted his life back', but now it appears he could be persuaded to stay after all.
He told the Radio Times: "I've been doing it ten years. It's an awful long time to be on the show. I don't want to be pushed. I'd prefer to leave than be pushed. To leave on my terms.
"Now if Cowell came back, I might come back. The two of us, we would not stop laughing. If Simon called me up and said… but, oh well. Let's see what happens."
Simon - who quit the show in 2010 to focus on 'The X Factor USA' - has been linked with a return next year after, after ITV pleaded with him to make a comeback as part of their on-going negotiations over the show's new contract with the broadcaster.
Gary Barlow is also thought to be departing after the upcoming tenth series finishes at Christmas to focus on Take That, paving the way for Simon's return.
However, his PR spokesman Max Clifford played down the reports to The Guardian, telling them: "Obviously, everybody to do with the show wants Simon because he gives it an extra quality, we understand that, but with his commitments over there he simply can't be in two places at once."