Conversion therapy
Mike Freer said: "When senior colleagues had throwaway comments, vilifying the trans community, that was simply not acceptable."
"You can cancel your conference, but you can't cancel us."
It will not be illegal for those aged over 18 to receive conversion therapy if they provide consent.
The issue has sparked fury from Tory MPs who have called on the government to "have some empathy".
Iain Anderson said the decision to exclude trans people from the conversion therapy ban was "deeply damaging".
Jamie Wallis's own government is pressing ahead with outlawing conversion therapy – but not for trans people.
The boycott was described as "hugely embarrassing" for the government.
The senior Tory was in India when news broke that the prime minister's was ditching a key pledge.
Boris Johnson was to u-turn on pledge to outlaw practice – but a furious backlash prompted a partial backtrack.
"A government that believes conversion therapy is acceptable in 21st century Britain is no friend of the LGBT+ community."