The willingness of US internet giants to co-operate with British law enforcement agencies in counter-terrorism operations and other serious cases remains "incomplete", a report to David Cameron has warned.
Sir Nigel Sheinwald, a former diplomat appointed by the Prime Minister to ensure UK police and intelligence agencies can gain access to data held overseas, said the current system was slow and bureaucratic and it could take months to deliver the information they needed.
He called for the creation of a new international framework on data sharing among "certain democratic countries" to enable them to request information direct from the companies in the most serious cases.
Since his appointment in September, Sir Nigel – a former ambassador to the US – said he had been working with the communications companies on the most urgent requests, particularly in counter-terrorism and other threat to life and child protection cases.
While the companies' assistance in these cases had improved, demonstrating the value of actively engaging with them, he said there was still more that needed to be done.
"Co-operation remains incomplete, and the companies and governments concerned agree that we need to work on longer term solutions," he said.
Sir Nigel said "limited and proportionate" access to private communication played a vital role in keeping the country safe - whether obtaining information about terrorist attack planning or locating a kidnapped child.
However law enforcement agencies seeking access to such material faced legal and technological obstacles while the disclosures by Edward Snowden about the activities of GCHQ and the US National Security Agency had provide a "challenging backdrop" to their work.
New companies were increasingly offering "end-to-end" encryption and storage of content on the user's own device rather than in data centres, making it harder to obtain data in a readable form.
In most cases the US Wiretap and Stored Communications Acts prohibited the disclosure of the content of electronic communications held or intercepted in the US while the US/UK Mutual Legal Assistance Treaty was seen as "slow, unresponsive and bureaucratic".
Hard copies of legal documents had to be couriered across the Atlantic through numerous intermediary bodies and it could take up to nine months for information to be returned.
While Sir Nigel said that the treaty could be improved, it would never be fast enough or have a wide enough scope for dealing with the most urgent cases.
He said that he had opened discussions with the companies and the US and other governments about a solution that would allow "certain democratic companies – with similar values and high standards of oversight" - to gain access to content through direct request to companies.
"This proposal offers a sustainable and longer-term solution to data sharing and would aid in resolving inter-jurisdictional issues," he said.
"Despite progress in improving short term cooperation, my work has reinforced the need for new longer-term, international arrangements, and more strategic relationships with the companies."