The former minister announced he would not be standing at the general election in his West Midlands constituency and advised people to back Boris Johnson.
“It really comes to something when I’m telling people to vote for Boris Johnson in this election,” he told BBC Radio 4′s Today programme.
But, he added: “Jeremy Corbyn is completely unfit to run the country.” He also told his local newspaper the Express and Star that “decent patriotic Labour voters” had no choice but to vote for the Conservatives.
Sign up now to get The Waugh Zone, our evening politics briefing, by email.
Austin, the adopted son of a Jewish refugee who fled the Holocaust, quit Labour in February over Labour’s anti-Semitism crisis.
He was elected MP for Dudley North in 2005, having served under Brown as press secretary. He went on to become minister for the West Midlands and assistant whip.
Reacting to his remarks, Labour’s shadow business secretary Rebecca Long-Bailey said: “Voting for Boris Johnson if you’re a Labour voter and you want to protect your community, that’s absolutely absurd.
“Jeremy is a patriot as we all are within the Labour party, we are very proud of our country. It’s wrong for him to suggest that.”
Austin has been sitting as an independent MP after refusing pleas to join the Independent Group of MPs earlier this year.
“I could not stand by as the Labour Party has been poisoned by racism, extremism and intolerance under Jeremy Corbyn’s leadership,” he said.
Asked if he was advocating for the electorate to vote for Johnson over Corbyn on December 12, he told the BBC: “I am.”
“Look, the public has to make this choice. The British people have to decide this.
“Lots of traditional Labour voters are going to be grappling with this question.
“If they have got to face up to that, then I don’t think people like me should have the luxury of running away from it.”
Austin also said the decision by deputy leader Tom Watson to quit parliament was “enormously significant”.
“If Tom thought that Jeremy Corbyn was fit to lead our country and fit to form a government, then he would have been in that Cabinet. Would he really be standing down?
“Anybody who has spoken to Tom knows what he thinks about Jeremy Corbyn.”