Goldsmith claims that the current bill wouldn't allow his constituents to kick him out even if he took a five-year holiday or defected to the British National Party.
The Recall of MPs bill, championed by deputy prime minister Nick Clegg, will face its first Commons test on Tuesday. It would allow constituents to hold a petition to remove their MP, but only under limited conditions.
Conservative MP Goldsmith said the bill would give the public no real power, because MPs could only be removed if they are sentenced to jail or the House of Commons decides they have committed "serious wrongdoing".
Research suggests few MPs would have been recalled if the proposed rules were in force currently.
Goldsmith and other campaigners for recall have claimed the bill is "a stitch-up" and say voters should be able to call a petition under any circumstances.
Days before the bill is discussed, he said that if it was passed, MPs could break "every promise" and not recalled by constituents.
"The mad thing is at the moment that I could, the day after the next election if I'm lucky enough to be re-elected, I could go off on holiday for five years without breaking a single rule.
"I could switch to the BNP, I could break every promise I made, I could do any of these things without qualifying for recall under the Clegg proposals and without being open to any kind of reprimand in the current system."
Goldsmith, a long-time campaigner for recall, added: "Anyone who understands what recall is, it's a very simple process. If enough people sign a petition in a constituency they earn the right to have a referendum, do they want to remove their MP or not.
"The Government Bill, the Clegg Bill if I can call it that, is almost the exact opposite so you will be able to remove your MP under the Clegg Bill but only with the permission of the House so the institution that this is supposed to be helping people hold to account will still be in charge, it gets to decide who qualifies for recall."
Goldsmith is attempting to amend the Government's bill and said it is so narrowly drawn it was "almost the exact opposite" of a true power of recall.
Under the proposed legislation, if an MP is sentenced to prison or is suspended, a petition will be open for eight weeks and if at least 10% of eligible electors sign it the seat will be declared vacant, triggering a by-election.
But Goldsmith has pushed to give constituents more power, with a rival plan which would see MPs being recalled if just 5% of voters in a constituency sign a "notice of intent to recall" and 20% then sign a "recall petition".
He told Sky News' Murnaghan programme that the 20% threshold, which would involve people having to physically sign a petition, was high enough to prevent "vexatious abuse ... but low enough to make it actually possible so if an MP really lets the constituents down, profoundly lets them down, it should be possible to remove them".
Prime minister David Cameron has suggested he is willing to take much further steps towards MP recall than the bill proposes.
Ukip's only MP Douglas Carswell - who defected from the Tories - challenged Cameron about the issue in the Commons last week.
Cameron told Carswell that powers in the Bill were the "minimum acceptable" and promised to look carefully at any amendments. "I think there are a lot of very good arguments to be had about how we can go further," he said.
Goldsmith said: "I thought the Prime Minister's answer to Douglas Carswell also indicated a willingness to really move considerably on this issue so it's good."
Goldsmith added that disillusionment with Westminster politicians had been a key part in the rise of Nigel Farage's party.
He said: "I am really grateful that the new Chief Whip (Michael Gove) seems to understand, more than any of his predecessors in my view that the Ukip factor is not about Europe, it's not about immigration, it's not about individual policies, it's about the sense that the political elite if you can call it that has become so remote that it needs a spanking, it needs a slap and Ukip is an option there, the Green Party is an option there but the mainstream parties are just operating as a block that ignores the voters, that's become so remote there is no point engaging with it.
"He gets that and I believe - I don't know this and nothing has been formally announced - but I believe when it comes to the debate on the recall of members, the Conservative Party at least will be offering a free vote, that's the understanding I have and I think that's a great indication that things are changing.
Goldsmith, whose father Sir James Goldsmith formed the Eurosceptic Referendum Party to fight the 1997 election, said he was not attracted to Ukip but could see why people turned to it.
But he said it would be a mistake for the Tories to try to ape their policies.
"I can understand why people are turning to these outsider parties, these challengers from the outside," he said.
"At one point the Lib Dems would have been that party, they are not any longer so it's Greens, it's Ukip and so on, I can understand why people are doing that.
"I think the big mistake that we could make would be to try and ape their policies in order to bring those people back because it's not about what we're saying, it's about the fact that people don't believe what we're saying, it's about authenticity."Suggest a correction