The actor, who joined the panel earlier this year along with Michael McIntyre, will leave after just one run of the ITV show, according to reports.
Show bosses said preparations for the next series are at too early a stage to confirm the line-up for next year. They are currently busy with ITV1's The X Factor.
A report in The Sun said that the former Baywatch star - whose ex-wife Pamela is in Channel 5's Celebrity Big Brother - had lacked "sparkle".
It also said comic Michael's position had been under threat because he was considered "too nice".
But a spokeswoman for the show said: "It is too early to confirm the panel for the next series of Britain's Got Talent."