The former ‘Strictly Come Dancing’ pro James lashed out the Hollywood actor after he asked James if he was gay, and then made a number of jokes about the dancer’s sexuality.
James was larking around in the bedroom with Leslie Jordan, who was going through The Duchess’s trunks while wearing a pair of white gloves when the incident took place.
When James jokily bent over and said he was “assuming the position”, Gary - in a seemingly serious tone - questioned his sexuality, leaving James explaining the difference between camp and gay.
Gary continued to pursue his line of questioning, asking James if he “liked to get butt-f***ed”, which is when the ‘Strictly’ star flew off the handle.
“I’ve never hit a seventy year old before, ever,” James said, adding: “It's normally the people like you - listen, don't be rude, look at me when I'm talking to you - who are thinking that homosexuality is wrong.
“You should be careful what you say, you may be 70 years old, but you should be careful.
“Do you think that's acceptable? You should have respect.”
Gary later put things right by apologising and explaining that he was joking, claiming he is still getting used to the British sense of humour.
The actor hasn’t had an easy time settling in to life in the ‘CBB’ house and later in the episode, viewers saw him pranked by a gang of the other housemates.
Gary was understandably less than impressed when Ricci snuggled up next to him, as the rest of the gang howled with laughter.
Despite his Hollywood success, Gary hasn't had the easiest time and in 1989 he was left fighting for his life after a motorcycle accident. Watch him discuss the near-fatal accident below...