He also said the polls should "never have been that close" and joked he would sue the polling companies for giving him nervous "stomach ulcers" in the run up to the vote last week.
The neutrality of the queen over Scottish independence became an issue in the runup to last week's vote.
After it was reported she was pro-unionist, she told people near Balmoral, her Scottish estate, to "think very carefully" before voting.
Cameron made the comments in New York to Michael Bloomberg, the city's former mayor, unaware he was being recorded.
"The definition of relief is being the prime minister of the United Kingdom and ringing the Queen and saying 'It's alright, it's okay'. That was something," he said.
"She purred down the line."
How embarrassing that our PM can talk about the Queen like that to a US media mogul surrounded by cameras. Idiot #purrgate— Alastair Campbell (@campbellclaret) September 23, 2014
What he says next is inaudible for a few seconds, before he added: "But it should never have been that close.
"It wasn't in the end, but there was a time in the middle of the campaign when it felt…"
After another few seconds of the conversation being inaudible, he said: "I've said I want to find these polling companies and I want to sue them for my stomach ulcers because of what they put me through.
"It was very nervous moments."
The incident triggered an immediate backlash on Twitter, where people variously mocked him while former Labour spin doctor Alistair Campbell called him an "idiot" and a journalist said an apology was "sure to follow".
PM caught by a mic saying Queen 'purred down the line' when he told her Scotland had voted No. Surely prime ministerial apology to follow— James Chapman (Mail) (@jameschappers) September 23, 2014
Taxi for Cameron. The Queen will not be amused but I am http://t.co/wn8jcEaFhO— Kevin Maguire (@Kevin_Maguire) September 23, 2014
David Cameron saying the Queen purred down the phone when he told her it was a No vote sounds like 50 Shades of Eton.— Scott Wilson (@HeartofFire) September 23, 2014
David Cameron overheard saying the Queen 'purred' when he told her the Scotland result. So she doesn't have a telly, then?— malc mcgookin (@malcmcgookin) September 23, 2014
Cameron says the Queen ‘purred’ on the phone. Are we sure he didn’t call George Galloway? #wouldyoulikemetobethecat— SimonNRicketts (@SimonNRicketts) September 23, 2014
"I am so relieved, Mr Cameron", the Queen purred softly, alive with regal joy.
He was strong, silent. He was still her Prime Minister.— Stig Abell (@StigAbell) September 23, 2014
I doubt the queen purred, she was probably just gargling on her teeth.— Hannah Kelso (@HannahKelso) September 23, 2014
A Buckingham Palace spokesman would not confirm or deny whether the Queen purrs when she is happy, Sky News reports.
The palace also said it would not comment on conversations between the prime minister and the Queen.
Buckingham Palace tell me: 'We would not comment on a private conversation between Her Majesty and the Prime Minister.'September 23, 2014
Journalists are pouring over the tape and anticipating a whirlwind of public damage similar to when Gordon Brown was picked up on a hot mic calling a Labour voter "a bigoted woman".
OMG- Cameron will be mortified by those comments in NY. I think the 'inaudible' bit is 'I've never heard someone so relieved' (the Queen)— Isabel Oakeshott (@IsabelOakeshott) September 23, 2014
David Cameron in extraordinary Royal gaffe after saying Queen 'purred' at him on phone http://t.co/cHZmBAJVNt— Sarah Knapton (@sarahknapton) September 23, 2014
I'd love to by fly on wall when @david_cameron next sees The Queen. 1. she already knew the result. 2. She doesn't purr. 3. Keep trap shut— Alastair Campbell (@campbellclaret) September 23, 2014