The TV watchdog received 32 complaints after the former rugby player was angered after contestants on the ITV reality show took too long to answer a multiple choice question.
In scenes that aired last week, James was seen snapping at his campmates as they debated which answer to choose as part of the Dingo Dollar challenge.
Annoyed at the length of time they were taking, he locked horns with Kate Garraway, before saying to Andrew Maxwell: “It irritates the piss out of me because it’s that simple. Explain the question, answer the question a or b – it’s a circus of stupidity.
“I can’t do it. I’m too intelligent and too switched on. Just put your hands right up, let someone count them, right the majority win, shut up and let’s move on.”
As is standard procedure, Ofcom will now assess the complaints before deciding whether to investigate.
Fans had also expressed their dislike of his remarks on social media, with the host of ITV2 spin-off show Extra Camp Emily Atack also speaking out against James.
“He’s one of those people that thinks ‘If you don’t think the same way as me, it’s incorrect.’ And that’s not fair,” she said on the show.
“He puts words in people’s mouths, like the fact that people want to showcase their talent - some people aren’t in there to do that.”
James previously prompted a flurry of Ofcom complaints after making a “clubfoot” joke on the show.
Some 78 viewers complained to the TV watchdog when, after seeing Ian running through the camp, James remarked: “He’s completely lost the plot. Look at that clubfoot. Go on then, expend that energy.”
James became the fourth celebrity to be voted out of the jungle during Tuesday night’s show.
I’m A Celebrity continues tonight at 9pm on ITV.