The unwritten rules of decorum state it is impolite to discuss sex, politics or religion at dinner parties. I would like to add one more topic to that list - cultural repatriation. As discursive stink-bombs go it's not often a headline act, but there are few controversies more likely to invoke a full-on food fight during the middle of the cheese course than the concept of returning archaeological heritage to various peoples around the globe. Now, just months from the Olympics, the campaign is being stepped up once more for the return of the Elgin Marbles to the Greek nation, and another messy argument seems inevitable.

First thing's first, why are they the Elgin Marbles? Well, here lies our first trip hazard - we do not refer to them as the Parthenon Marbles (the building they were intended for) or the Phidian Marbles (the sculptor who crafted them), but instead they have taken the name of the aristocrat who nabbed them from Greece. As far as I am aware, lumps of rock are unaffected by Stockholm Syndrome, so it's not the Marbles themselves who are identifying with their kidnapper. No, it's the British people who have dubbed them Elgin's Marbles, in gratitude for the Lord's generosity in selling them, at a reduced price, to the nation in 1816. So, already Britain has committed an act of appropriation through nominative rebranding. The name implies they were Elgin's to sell in the first place.

This is a complex issue, more complex than a Game of Thrones boxset that has been accidentally translated into Kurdish. Lord Elgin was Britain's ambassador at the time, but not to Greece. Instead, he was diplomatically engaged with the Ottoman Sultanate which had ruled Greece since the 15th century and was therefore in charge of its heritage. The Sultan was grateful for British support against Napoleon Bonaparte, allowing Elgin to secure a highly dubious (and still contested) legal writ permitting him to remove the sculptural treasures from the famous Parthenon, which sits proudly atop the Acropolis in Athens. His initial idea had been to simply take drawings and plaster-casts, but the British government had responded with all the enthusiasm of a dead badger. After learning that local peasants had been chiselling off these marble masterpieces and burning them to make quicklime, Elgin decided he would ship them instead to a place of safety. In his mind, that could only be the British Museum. True to his word, he refused to sell them to private bidders (or to Napoleon) and sold them at a loss to the BM.

This was a controversial act, even in Regency Britain. Elgin did not receive universal acclaim for his forward-thinking rescue mission. Lord Byron, who would soon-after die abroad while supporting Greek independence, found it to be an act of tyrannical vandalism, and did what any furious politician would do... he wrote some poetry about it. Somewhat ironically, Elgin's Marbles - which were 2000 years old and beautifully carved panelled reliefs by one of history's greatest ever sculptors, Phidias - were dismissed by some art critics as worthless, because they were mucky and partially damaged. Apparently, ancient art had to be anatomically correct if it was to be worth keeping. However, despite the kerfuffle on two fronts, the marbles were adopted into the British Museum's collection and have stayed there ever since.

Scroll forward to 2012, and the Olympics are almost in town. Once again, the bandwagon for repatriation has been rolled out, though this time it will have to pay the London congestion charge first. This week, Stephen Fry has joined many others in calling for the full repatriation of the Marbles to Athens. It seems an entirely fair request. Since Elgin's visit, Greece has matured as a nation, going from a bankrupted suppressed province to a bankrupted sovereign state... Okay, so it is not exactly in rude health, but the explosive days when the Parthenon was used as an ammunition dump (it blew up in 1687) are long gone. Today, Greece is genuinely passionate about its heritage.

However, there are still some niggling doubts being aired. For one, the Greek financial apocalypse has irradiated much of its heritage budget, and several ancient sites are already under threat from lack of conservation investment. Using an analogy from children's social services, there are those who claim that Greece is an unfit parent and cannot be trusted with such a vulnerable offspring, who has already been blown up once in the past. This argument, to me, seems unfair. Arguably, returning the Marbles to the Parthenon would increase tourism, and provide a fresh influx of cash to pay for their protection. At present, the museum in Greece has a dodgy plaster-cast facsimile. How would we feel if we travelled to Salisbury Plain only to be confronted with Foamhenge, because Stonehenge were in Abu Dhabi for its own protection?

The greater debate, however, is one of precedent. If we return the Elgin Marbles, then suddenly Britain's entire heritage collection becomes fair game. Egyptian antiquities, Renaissance paintings, Roman busts, Indian treasures, Chinese vases - many of the collections in our museums, and most famously the British Museum, were acquired during the height of Empire, when we straddled the globe. China was far from a superpower then - instead it was a victim of western colonial meddling, resulting in two Opium Wars and the Boxer Rebellion. Today, however, China bankrolls the west - if it started asking for stuff back, like an angry girlfriend who realises she has wasted her best years with an idiot, our only defence is a blanket policy of refusing repatriation. Returning the Marbles to Greece would shatter that get-out clause.

But there is a further, more nebulous issue at stake here. What is culture, and can it really be nationally ascribed? Can we really state that every artefact is morally tied to the place in which it was created? Take for instance a Celtic torc - a thick gold necklace worn commonly in the Iron Age - these are objects that spanned geography. The Celts did their thang throughout Europe, from Greece to Ireland. If archaeologists excavate a priceless torc in France, but experts say it was made in Germany, then who gets to keep it? Can Ireland put in a claim for it, under the argument that it is representative of their national heritage? Should it simply tour around Europe forever, like the Rolling Stones?

This is a grey area larger than the smog over China's cities. To take Britain as an example, we are a nation founded on a fusion of Celtic, Germanic, Scandinavian, French, Dutch, and African peoples, to name just a few. Our political system is predicated on Ancient Greek notions of democracy, while our law is influenced by Rome's Twelve Tables. Our religious history shares commonality with the Middle East, and our intellectual heritage owes a great deal to Renaissance Italy. To further confuse things, it was our men and women who founded modern America and Australia, and it was our engineers who designed railroads in India. Under these tortuous conditions, it's easier to convert Sarah Palin to communism than it is to neatly identify Britishness.

In fact, perhaps we shouldn't even want to, as it's a dangerous game claiming archaeological heritage for nationalist reasons. Though I usually hate the tired analogy of 'Hitlerism', we should always be mindful of the ways in which Gustav Kossina and Carl Schuchhardt used archaeological theories to justify fascist ideology in Germany. Their interpretation of Nordic/Aryan culture in surrounding countries was used by Hitler to justify his policy of lebensraum land-grabbing, under the pretext that Germania had once been far larger, and he was simply restoring the natural order of things. You can get into all sorts of trouble trying to marry up historical culture with contemporary political identity.

This is not to say the Greeks are fascists! Fear not, I don't plan on turning into Rush Limbaugh and embarking on a deranged rant. Personally, I would like to see the Elgin Marbles loaned back to Athens for a trial period, so they can at least be briefly seen in their natural setting. My reasons are relatively simple. Nearly 3,000 years ago, Greece invented the Olympics as a mono-cultural religious festival of athleticism - only Greeks could partake back then, yet here we are today about to celebrate the Olympiad in London. The Olympics have become universal heritage, originating in Greece but shared by the world. I do not see how the Elgin Marbles can be excluded from a similar position, and for one nation to steadfastly refuse to share them is naughty. Yes, it's true that tourists can come and view them for free in London, but not everyone can afford such a trip.

The Olympic Games travels the world, arriving in distant cities and bringing its ideals and history to faraway doorsteps. It inspires in places where inspiration may previously have been lacking, and it unifies millions of strangers who appear to share nothing in common. Similarly, the Elgin Marbles are some of the finest ancient treasures in the world, and reveal the origins of several artistic traditions. They too possess an inspirational quality, and a pan-global legacy. It would be rather wonderful if, just like the Olympics, they could travel to the people, not the other way around.

Perhaps Mick Jagger wouldn't mind taking them with him, on his next tour?

Follow Greg Jenner on Twitter: www.twitter.com/greg_jenner