More than 80% of Ukip supporters say they are uncomfortable with the number of people from ethnic minorities living in Britain, a poll has found.
And supporters of Nigel Farage's party were also more likely to be uncomfortable with a non-white babysitter.
After Lord Heseltine made the accusation last week, Farage responded on Sunday, saying Ukip was not racist but instead was about "standing up for this country".
One expert in the field told HuffPost UK that despite Ukip supporters' feelings on minorities, this did not make them racist.
British Future director Sunder Katwala said: "I think there's a discomfort, but that is not the same as racism.
"My view is that they are very different to the EDL or the BNP, but that they appeal to people who are uncomfortable with the speed of change in society.
"I think they are not a racist party, but they should continue to police the boundary against racists and extremists."
The pollsters told people about estimates that 10% of people in Britain were from an ethnic minority.
When asked how uncomfortable or comfortable they were with this, 83% of Ukip voters were either 'very' or 'fairly' uncomfortable.
By contrast, 50% of Tories said this, 38% of Labour supporters and 22% of Lib Dems.
A total of 18% of people would be uncomfortable with a non-white babysitter, the poll found, but this rose to 32% among Ukip supporters - compared to 22% who said they would be comfortable with one.
All parties have extreme voters, Katwala said, but Ukip supporters' responses did not make them racist.
He added: "For a long time, they have been called racist. They should be called racist if they say something racist - admitting to discomfort is not the same."
The findings came from a poll that also found 34% of the public would be uncomfortable with a Prime Minister who was from an ethnic minority.