Labour needs to do more to reassure voters it understands their concerns about immigration and welfare, former shadow cabinet minister Chuka Umunna said.
The ex-shadow business secretary also said immigrants have to do more to integrate with the local community, claiming it is "crazy" for someone who has been in the country for 30 years not to speak English.
He said Labour must recognise concerns about welfare, including acknowledging some people try to "rinse" the system.
At a fringe event at the Labour Party conference in Brighton, Mr Umunna - whose father was a Nigerian immigrant - said: "Immigration clearly has been an issue that our movement hasn't felt terribly comfortable talking about. Actually, I don't think we should have an inhibition about talking about immigration at all."
He said some recent migrants from Somalia, Eritrea and Eastern Europe are "living parallel lives" and not mixing with local communities.
"My view is that integration is a two-way street. Both the settled, existing population and newcomers have a responsibility," he said.
"I do think it is crazy that we do have first generation immigrants in our country who have been here for 20 or 30 years who can't speak English."
Being able to speak English helps "barriers break down" and can take some of the "toxicity" out of the issue, he said.
But Mr Umunna warned: "The idea that we should in any way buy into the narrative that says 'immigrants are responsible for all of our people's problems' and go down any avenue which seeks to set up different groups against each other, we should have a firm line that we are not going anywhere near that at all."
In a sign that Labour has to "meet the public where they are at", Mr Umunna said: "We have got to get into this conversation, in the same way that we have got to get into the conversation around our social security system.
"I have had young people come to my surgeries who can't find work, they don't want to be just doing nothing where they live all day long, they want to be working.
"But equally we all know there are people who rinse the system as well.
"There has got to be a way for us to get into that debate in a way that doesn't sound like we are being defensive, that we go on the offensive, that we are for the people who want to work hard and we are also for maintaining the integrity of our welfare state."
Mr Umunna, who refused to serve in Jeremy Corbyn's shadow cabinet because of major policy differences, insisted he has "no interest in being a thorn in the side of the leadership".
Asked whether he believes Mr Corbyn is the "best possible person" in Labour to be the next prime minister, Mr Umunna replied: "I want the Labour Party to win the next general election. I want whoever is our leader to be prime minister."
He criticised Ed Miliband's leadership, complaining that "our message changed every single bloody month".
He mocked the focus on the "cost of living crisis", adding: "Who wants to be told they've got a crisis? They want to know what the solution is."
He continued: "We changed our message constantly. No wonder nobody knew what we really stood for."
Mr Umunna pulled out of the race to succeed Mr Miliband just days after announcing his intention to stand, blaming intrusion into his family life. Explaining his short-lived campaign, he said: "I had a situation where my political life, my work, directly contradicted with the rest of my life and my loved ones and my family.
"Too often in the past I had generally tended to put the Labour Party and my work as an MP first but I thought for once, actually, I wouldn't be being true to myself or my family if I didn't withdraw."