Spy agencies must retain the power to collect communications data in bulk but the law governing online surveillance is "undemocratic" and must be entirely rewritten, a major report has found.
Terror laws watchdog David Anderson QC called for security services and police to keep intrusive powers to combat terrorism and serious crime but urged the Government to draw up "comprehensive and comprehensible" new legislation "from scratch".
In a 372-page landmark review, he said "bulk collection" of intercepted material must be allowed to continue - provided additional safeguards are introduced - citing a number of cases in which it had been used to thwart terror plots.
Another key recommendation would see the authority to sign off interception warrants, which allow agencies to view or listen to the content of a suspect's communications, transferred from ministers to judges in most cases.
A "law-based system" must also be developed to cope with the emerging problem of encrypted services, it said, warning that "no go areas" that cannot be accessed by intelligence services must be minimised.
Forcing technology firms to retain web browsing histories - a key plank of the controversial legislation labelled a "snooper's charter" by critics - may be "useful" but a "detailed operational case" must be made out, the review said.
Home Secretary Theresa May announced that draft plans for new surveillance powers will be published in the autumn.
Civil liberties groups and privacy campaigners welcomed calls for an overhaul of laws while police leaders insisted new laws should strengthen their access to communications data.
Mr Anderson, the independent reviewer of terrorism legislation, was commissioned to carry out the review outside of his normal duties when the Government rushed through new laws to maintain existing capabilities last year.
The report, titled A Question Of Trust and containing more than 100 recommendations, concluded that the Regulation of Investigatory Powers Act, which was introduced 15 years ago to govern surveillance techniques, has been "patched up so many times as to make it incomprehensible to all but a tiny band of initiates".
It added: "A multitude of alternative powers, some of them with statutory safeguards, confuse the picture further.
"This state of affairs is undemocratic, unnecessary and - in the long run - intolerable."
Speaking today, Mr Anderson called for a "clean slate".
He said: "Modern communications can be used by the unscrupulous for purposes ranging from cyber-attack, terrorism and espionage to fraud, kidnap and child sexual exploitation.
"A successful response to these threats depends on entrusting public bodies with the powers they need to identify and follow suspects in a borderless online world. But trust requires verification."
Reports have suggest encrypted services are increasingly being used by criminals and terrorists to avoid detection.
Mr Anderson said authorities should not be able to "access everything" but warned encryption is one of the reasons why "life is getting more difficult for them".
Intelligence agencies do not want a "permanent trump card" but do seek cooperation from companies abroad as well as in the UK, the report said.
Mr Anderson suggested a "law-based system" in which encryption keys may be handed over after "properly authorised requests".
Insisting that "no go areas" must be minimised, he said: "I just don't think you can have channels of communications that everybody knows, with absolute assurance, can never be penetrated. If that is the case, every paedophile network is going to use them, every terrorist network is going to use them, and you have effectively abdicated the rule of law."
Accessing these types of communications should be "hard" for authorities, he said, adding: "It is not the same as saying that every word that is said should be easily readable by the authorities."
He said the authorities were engaged in a "cat and mouse game" in which neither side has a "guaranteed upper hand".
Another key proposal concerned the introduction of judicial authorisation of all warrants for interception of data, which are currently signed off in all cases by the Home Secretary.
A total of more than 2,000 warrants were issued last year.
Mr Anderson said: "Do we really want a Home Secretary who has to authorise 2,345 of these warrants in a year as well as managing a huge department of state?"
His report said powers to force firms to retain data about the context but not the content of customers' communications for up to a year should remain in force.
Forcing firms to keep internet browsing records known as "web logs" would be "useful" for attributing communications to individual devices, identifying use of communications sites and gathering intelligence or evidence on web browsing history, Mr Anderson wrote.
However, the report stressed: "If any proposal is to be brought forward, a detailed operational case needs to be made out, and a rigorous assessment conducted of the lawfulness, likely effectiveness, intrusiveness and cost of requiring such data to be retained."
It also called for the security and intelligence agencies including listening post GCHQ to continue to be able to practise "bulk collection" of intercepted material and associated data.
Mr Anderson said he had been "persuaded" bulk collection is "very useful", adding: "It is a huge power, and that is why it's important to have maximum regulation and transparency."
The report cited a number of examples when the technique has been crucial to terrorism investigations.
In one case last year, bulk data analysis of known Islamic State, or ISIL, extremists sparked an international investigation which ultimately prevented a bomb plot in mainland Europe.
However, this capability can only be maintained if strict additional safeguards are introduced, such as limiting the ability to apply for "bulk warrants" to the heads of MI5, MI6 and GCHQ, the report said.
Mr Anderson said the "threat" of ignoring his recommendations was that the public becomes "disenchanted" and believes some of the "wilder allegations" that emerged from the Snowden affair.
National Police Chiefs' Council chairwoman Sara Thornton said officers "simply do not have the coverage which we had five years ago", adding: "We support David Anderson QC's finding that there should be 'no no go areas' for those charged with law enforcement because without access to communications data we are less able to keep people safe."
Shami Chakrabarti, director of Liberty, said: "Whilst we don't agree with all his conclusions, Mr Anderson's intervention could be the beginning of re-building public trust in surveillance conducted with respect for privacy, democracy and the law. It is further vindication of Edward Snowden's courage."