Owen Smith has launched a fresh attack on Jeremy Corbyn for failing to crack down on anti-Semitic and misogynistic abuse within the Labour Party.
In the latest leadership debate, Mr Smith said they should be "deeply ashamed" that they were mired in controversy about its attitude towards women and Jewish people.
Mr Corbyn insisted he had dealt "very robustly" with complaints of bullying and abuse and said he wanted to "reach out" to MPs in the party who had opposed his leadership.
However, Mr Smith warned there was a "chasm" between the faith that many party members had in Mr Corbyn and the majority of MPs.
Appearing on BBC Two's Victoria Derbyshire show, Mr Smith said Mr Corbyn had not stamped down hard enough on abuse within the party.
"There have been occasions when Jeremy has just not seemed bothered enough by it," he said.
"Lots of my female MP colleagues feel Jeremy wasn't supportive enough. Lots of the Jewish MP colleagues feel Jeremy wasn't quick enough to condemn anti-Semitic abuse.
"We are the Labour Party and we are having a debate on national television about anti-Semitism and misogyny within the Labour Party and that should make us all deeply ashamed.
"There are shades of hard left opinion that have drifted into anti-Semitic abuse about Israel and that is at the heart of the concerns that many in the party have."
Mr Corbyn insisted that he had made clear that such abuse had no place in the party and that the rules were being tightened.
"I have dealt with very strongly and very robustly any complaints about abuse. It has no place in any political party. I have made absolutely clear that any kind of abuse is totally unacceptable in any form," he said.
He said that if he was re-elected as leader he would invite critics who walked out of the shadow cabinet to re-join.
"I wanted to create a broad-based shadow cabinet, a broad-based team and I will do exactly the same again if I am re-elected in September," he said.
"I would ask those Labour MPs to remember that they have been elected - like I have, like Owen has - as Labour MPs.
"We have a huge responsibility to represent our party and the people that have elected us in Parliament and that means we have got to work together."
The pair clashed repeatedly over the issue of abuse within the party.
A young Labour supporter said she would "feel more comfortable" going to a Conservative Party conference and saying she was a Labour supporter than going to a Labour conference and saying she supported Owen Smith.
Mr Smith said the reason there is abuse within the party is that "we have become divided between people who are supportive of you and the rest of the Labour Party" with supporters of Mr Corbyn feeling he is the "only socialist in the Labour Party".
"The truth is I'm not a red Tory, I'm not a Blairite, I'm a socialist same as you," Mr Smith said.
The leadership-hopeful said that people within Labour who do not support Mr Corbyn are deemed "not pure enough".
But when Ms Derbyshire suggested to the Labour leader that the atmosphere within the party had become "toxic", Mr Corbyn hit back and asked: "Well, how do you know?"
Mr Corbyn said he had attended Labour events all over the country with "people of all shades of opinion having an intelligent, respectful discussion and debate".
"That is how we should do things at all times," he said.
Ms Derbyshire then asked the audience if they believed the atmosphere within the party has become "toxic", prompting a loud cheer of "yes".
Mr Smith also attacked Mr Corbyn over Labour's record in opposition.
He said the party has been "incredibly weak" under Mr Corbyn as he claimed responsibility for some of the party's recent victories in Parliament, including defeating the Government over proposed cuts to disability benefits.
"I won that for the Labour Party," Mr Smith said to which Mr Corbyn replied that "we" had won.
Mr Corbyn was asked how successful he has been at "reaching out" to members of the parliamentary Labour party who disagree with him.
He said: "It can work on certain occasions. It can work very well."
Mr Corbyn was then pushed for an example.
He said: "I have worked very well with Andy Burnham since he became shadow home secretary... I was very happy to work with Owen on the issues that he was responsible for, I'm very happy to work with John McDonnell on the issues he is responsible for."
The mention of Mr McDonnell, one of Mr Corbyn's top allies, prompted outcry from a section of the audience.