Liberal Democrats are calling on David Cameron to ditch his election campaigner following the decision to shelve plain cigarette packs.
Former health minister Paul Burstow told The Observer Crosby should either quit or be sacked because of his links with the tobacco industry.
It has emerged the Australian strategist's lobbying firm, CTF, had worked for Philip Morris, one of the Big Four tobacco companies, giving it advice "on a range of matters".
Speaking on Sunday morning, Tory Party chairman Grant Shapps described the allegations as a "smear campaign".
Crosby advises the Tories on election strategy
He told Sky News no policy could be decided without the Lib Dems' involvement, and said Crosby advised the Conservative Party, not the Prime Minister.
It is "impossible" for Crosby to have an input on government policy, he said, claiming criticism from Labour was "an attempt to divert attention".
The government has delayed plans for plain cigarette packets, saying it wants to wait for hard evidence from Australia before making a decision.
This has led to criticism from public health campaigners, and MPs from all parties.
Shapps insisted: "This is about getting the evidence before you decide to do something."
But Stephen Williams, another Lib Dem MP and chairman of the all-party group on smoking and health, also called for Crosby to go, saying there was "a massive conflict of interest".
He added: "The prime minister should be considering whether he believes Crosby's position is tenable."