Almost a fifth of England's most precious historic buildings were hit by crimes ranging from vandalism to metal theft last year, English Heritage said.
The first comprehensive survey of the impact of crime on England's historic buildings and sites found that churches and other religious buildings were the most affected, with more than a third (37.5%) damaged by criminal activity.
In total some 70,000 listed buildings, accounting for 19% of the total listed building stock, were hit by crime, with almost half of those badly damaged, the report said on Monday.
Structures with the highest level of protection were the worst hit.
Metal theft in the face of rising global prices is the single biggest problem facing old buildings, particularly places of worship, with one in seven churches damaged by having materials such as lead stolen from them last year.
In one case reported in the survey, a church in Hampshire saw its organ ruined by water leaks after thieves repeatedly stole metal from the roof.
For scheduled monuments such as prehistoric stone circles and archaeological sites, the biggest problems are anti-social behaviour including flytipping and trail-biking, which can deter visitors and investment, and illegal metal detecting for treasure.
Protected wrecks are also subject to criminal activity from illegal diving to theft, with a case in the report in which several 16th century bronze cannon, metal ingots and Second World War ammunition were seized by the authorities.
Other examples of crimes against England's heritage include a group who drove a 4x4 over the remains of a Roman settlement and a man who spray-painted the internationally important Clifford's Tower in York - with details of his Twitter account.
The damage caused - which can in some cases never be put right - can run to hundreds of thousands of pounds in the rare cases of arson, and tens of thousands of pounds in the case of metal thefts from buildings, the report suggests.
Damage to listed phone boxes alone cost BT £120,000 in a year, the report found.
Dr Simon Thurley, chief executive of English Heritage, said: "Whilst heritage is not necessarily being targeted over other places, save perhaps for their valuable materials and artefacts, they are suffering a substantial rate of attrition from crime nonetheless.
"Damage done to a listed building or an archaeological site can often not be put right and centuries of history will be lost forever. These places have an obviously high value to society.
"Their particular vulnerability warrants every effort to ensure they are still around for future generations to enjoy just as much as we enjoy them now."
According to English Heritage's legal director Mike Harlow, heritage buildings can often be caught in a spiral of decline when they are subject to criminal damage, which in turn affected their surroundings.
He said a building which becomes unused can be subject to minor vandalism and graffiti and then, when people realise it is not being looked after, they may strip materials, letting the rain in and damaging the structure, putting the whole building at risk and making people feel like the surroundings are a no-go area.
"If you make the building looked after, it creates a more comfortable environment," he said.
"This isn't just about saving heritage, it's also about making safer places that feel welcoming."
English Heritage has been running a heritage crime programme for two years in a bid to reduce criminal activity against listed buildings and monuments, including introducing impact statements in court cases to explain the effects of damage.
An impact statement in the case of the man who graffitied Clifford's Tower led the judge to hand down a custodial sentence.
Harlow also called for younger offenders in particular to have relevant non-custodial sentences, for example being made to clean up the damage they had done and learn about why the heritage was important.
A network of organisations including charities, local authorities and local groups are working with English Heritage, the police and the Crown Prosecution Service to raise awareness of the issue and take steps to reduce crime.
In one initiative, 150 at-risk Cornish medieval wayside crosses have been microchipped to prevent them being stolen.