The controversial MP made the comments after lawyers acting for the men launched legal action to prevent them being put onto the Bibby Stockholm vessel.
Around 15 men finally boarded the barge yesterday after weeks of delays, but 20 objected to being removed from hotels to live there.
Anderson, the MP for Ashfield, told the Daily Express: “If they don’t like barges then they should fuck off back to France.”
He said: “Lee is expressing in salty terms I think something that people well understand, which is that the British people have warm hearts, we also want to have a secure front door and it is not right that people that behave illegally ... to jump the queue compared to those who play by the rules.”
Robinson replied: “Should they f-off back to France as he suggests?”
Chalk said: “Lee has expressed himself in his own way.”
Undeterred, Robinson said: “Many people listening don’t know you well, you became justice secretary relatively recently replacing Dominic Raab, you said a few years ago that you believed Britain was a ‘tolerant, outward-facing nation’ and you denounced people using rhetoric you didn’t like as ‘an excuse for ugly bigotry I thought we’d left long behind’.
“Do you think telling people to f-off back to France who might have genuine claims to be asylum seekers, who might have escaped war and persecution, do you think that is the reflection of a tolerant, outward-facing nation?”
But Chalk insisted he had “no difficulty” with what Anderson had said.
“It’s not bigotry at all,” he said. “This country has offered its home to 400,000 people since 2015 – Ukrainians, people from Hong Kong, Afghans.
“We are an open and warm and outward-facing country. But equally, we are a country that believes in fairness and playing by the rules and those who don’t play by the rules, who don’t take advantage of those opportunities that we provide, and try to jump the queue, rightly cause indignation.
“I think that Lee was expressing that indignation in his own way and there was nothing unreasonable in principle with that he was saying.”
The clash came amid the ongoing controversy over the use of the Bibby Stockholm to house migrants.
Ministers insist the move is necessary to reduce the £6 million-a-day cost of putting them up in hotels.
The men refusing to board the barge are being supported by the Care4Calais campaign group.
Chief executive Steve Smith said: “Amongst our clients are people who are disabled, who have survived torture and modern slavery and who have had traumatic experiences at sea. To house any human being in a ‘quasi floating prison’ like the Bibby Stockholm is inhumane.”