David Cameron has indicated Lord Sewel should be kicked out of the House of Lords over "very serious" allegations he took drugs with prostitutes.
The Prime Minister said further questions will be asked about whether it is appropriate to have someone involved in passing laws if they have "genuinely behaved this way".
Scotland Yard has been called in to investigate the former minister after The Sun on Sunday published video allegedly showing him snorting cocaine while romping naked with two women.
The peer quit his £84,500-a-year role as deputy speaker of the Lords, which included overseeing conduct issues in the Upper House, after the allegations emerged and has since been suspended from the Labour Party.
Speaking to reporters in Jakarta, Mr Cameron said: "These are very serious allegations. I think it's right he has stood down from his committee posts and I'm sure further questions will be asked about whether it is appropriate to have someone legislating and acting in the House of Lords if they have genuinely behaved in this way.
"It's still going to take some time I suspect to get to the full truth."
The newspaper's footage shows the peer snorting white powder - alleged to be cocaine - from a prostitute's breasts using a £5 note.
He is also pictured wearing an orange bra and leather jacket as he reclines smoking a cigarette.
The 69-year-old apparently paid one of the women for the night with a cheque for £200, dated July 22.
In a conversation reportedly recorded in Lord Sewel's flat in Dolphin Square, Pimlico, a couple of miles from Parliament, the peer branded Mr Cameron "the most facile, superficial prime minister there's ever been".
He labels Mayor of London Boris Johnson "a joke" and a "public school upper class twit", and describes SNP MP Alex Salmond as a "silly, pompous prat".
Lord Speaker Baroness D'Souza branded the married father's behaviour "shocking and unacceptable" and said she was referring him to the police.
"These serious allegations will be referred to the House of Lords Commissioner for Standards and the Metropolitan Police for investigation as a matter of urgency," she said.
The House of Lords (Suspension and Expulsion) Act 2015 - which only received Royal Assent in March - allows peers to be barred from parliament if they breach a beefed up code of conduct.
Earlier this month, Lord Sewel wrote an article on the new rules that stated: "Scandals make good headlines. The requirement that members must always act on their personal honour has been reinforced."
Police sources indicated they would assess the evidence after the referral from Baroness D'Souza, but pointed out that drug-taking allegations were notoriously difficult to prove when there was only video evidence, rather than substances that could be tested.
In between apparently snorting lines, Lord Sewel is heard complaining that he struggles to afford the £1,000-a-month rent on the flat.
The peer is asked whether he receives expenses and explains that he now gets a flat-rate allowance of £200 a day. "It's all changed and disappeared. People were making false claims," he said.
"Members of her Lordship's House who are right thieves, rogues and bastards at times. Wonderful people that they are."
In fact, the per diem for attending the Lords is £300, and it did not apply to Lord Sewel. As he declares his main residence is in Aberdeen, he was entitled to a tax-free office holder's allowance of £36,000 a year.
Lord Sewel remained a member of the Labour Party despite giving up the whip under parliamentary conventions when he became deputy speaker.
A Labour spokesman said: "Lord Sewel has been suspended from the Labour Party."