“I was quite honoured to be honest,” he told Capital FM about the rumours. “It's a big job. Would I say no? I don't know. I haven't been offered anything.”
However, despite being up for the job, he added it would depend on what his boss Simon Cowell would have to say about him judging The X Factor's biggest rival.
“To be honest, I don't think it's my decision,” he admitted. “I think it will be more down to Simon and The X Factor's decision if they let me do it.”
'The Voice' coaches
He added: “I don't think I want Simon Cowell as an enemy – so I don't think that's probably going to happen.
"If there was an offer on the table, you'd definitely consider it.”
Olly isn't a stranger to TV, having hosted two series of 'The Xtra Factor' with Caroline Flack.