Sean Worth, who spent just over two years as a special adviser to prime minister following the general election, said on Wednesday that voters still thought the Tories had a bad "smell" and did not trust the party on issues such as education and healthcare.
"When we talk about social policy, when our political leaders talk about it in the Tory party, it's often in the language of sympathy and 'oh I'm so sad about all these terribly poor people and the plight that they're in' and 'of course we are very concerned about it all'," he said. "People just don't buy it, there is still that smell."
Worth added: "What I would like to see is the Tory party replacing that with some radical things and tough ideas."
The former Downing Street aide who now works for the Policy Exchange was speaking at the official Westminster launch of the modernising Tory Bright Blue think-tank. The group, backed by several MPs including former defence secretary Liam Fox, advocates a progressive liberal form of Conservatism.
Worth, who penned a chapter in the think-tank's new Modernisers Manifesto book, said Conservatives needed to take on Labour on issues had for too long been seen within the party as "wet".
"I think we need to see a lot less sympathy and simpering from the Tory part on areas where it doesn't feel confident enough to speak," he said. "Issues like the Big Society and localism are purely theoretical, we need to talk much more toughly about hospitals, schools and care."
Before working for Cameron in No.10 after the election, Worth was head of the Conservative Party's Policy Unit. And he told the room of Tory activists that the party had not done enough to connect with "ordinary people" during the 2010 campaign.
"I would go round talking about the policies we were pushing out on the doorstep and they would just bomb. they wouldn't cut through to people," he said.
Worth added: "It's about making the Conservative Party, stop simpering and waffling on about things like the big society, and take on vested interests on behalf ordinary people ... you have to take a bullet for those people for them to believe you as Thatcher did and Blair did to an extent."