They say there is no evidence the move would raise maths standards and could even be a 'backward step'.
The warning comes as thousands of Year 6 pupils (aged 10 and 11 years) prepare to sit maths tests.
But Schools Minister Elizabeth Truss said children needed to be confident with maths skills 'before they pick up a calculator'.
Until this year, the Key Stage 2 national curriculum tests, often known as SATs, taken at the end of primary school included a mental arithmetic paper, one maths paper where calculators were used and one where they were not allowed.
But in November 2012, Ms Truss announced that calculators would be banned in maths tests at Levels 3 to 5 from this summer.
However, calculators can be used for Level 6 papers which are sat by a small number of high-achieving pupils. The limit on the use of calculators was intended to make sure that children learned maths skills for themselves rather than relying on the help of a calculator.
But academics at Oxford University, Cambridge University and King's College London have challenged the principle that stopping the use of calculators will be beneficial in maths lessons.
Anne Watson, emeritus professor of mathematics education at Oxford University, said: "There is a substantial amount of good evidence on calculators in schools, mainly from the US, and none of it shows their use is detrimental to pupils' learning.
"In fact, students who use calculators regularly in lessons score as high or higher in tests, taken without calculators, compared to those who do not.
"On the whole, the use of calculators as an integral mathematical tool has been shown to be beneficial, particularly in the development of mathematical problem solving. It is a pity that current policy is retrogressive in this respect."
Ken Ruthven, professor of education at Cambridge University, said the use of calculators could enhance children's mathematical capability.
"As well as making calculation more efficient and reliable, calculators allow people to tackle mathematical problems in new ways.
"Making intelligent use of tools such as these underpins a great deal of the mathematics that is done in our contemporary world."
But the Government has rejected the argument, saying that in high-performing education systems such as Singapore, Hong Kong and Massachusetts in the US there is a recognition that 'calculators should not be used as a replacement for basic understanding and skills'.
Ms Truss said: "All children should be confident with addition, subtraction, times tables and division before they pick up a calculator.
"It is vital that children have a solid grounding in the basics so they can grow up to be comfortable with the maths they need in their adult lives. Banning calculators in primary school tests will help end the culture of reaching for a calculator at the first sign of a tricky sum.
"Some of the world's top education systems already do this and there is no reason why children in England can't compete with the best. Ensuring children leave primary school with a strong grasp of mathematics is a vital part of a long-term economic plan to safeguard this country's future."
More on Parentdish: The trouble with maths