It has been revealed that over one million viewers have turned off since the chef quit the BBC One cookery show back in March.
A host of guest presenters have been drafted in to host since James' exit, but it seems fans have been left decidedly unimpressed with their stints at the head of the show.
The Sun has reported that the most recent episode, hosted by Donal Skehan, saw just 1.4m tune in, compared to the 2.4m who watched James' farewell.
A source added to the paper that the guest hosts were "not working out as well as bosses had hoped".
The likes of The Hairy Bikers, Lorraine Pascal, Michel Roux Jnr have all taken the reins of 'Saturday Kitchen' since James' decision to quit after 10 years of hosting.
After various reports speculated about the motivation behind his departure, James spoke out to set the record straight.
"That’s every Friday and Saturday you’re giving up. I had to be in bed by 9pm and up at 3am. It’s a tough show: there’s a lot of content and when that red light comes on it’s just you, cooking and interviewing, with 26 people waffling in your ear."
He added: "I didn’t think leaving 'Saturday Kitchen' would be such a big deal. I genuinely said to my agent, 'Stick it out there, but no one will be bothered,' and next minute it was all over the papers. It was like One Direction splitting up."
'Saturday Kitchen' airs on Saturdays at 9am on BBC One.