Allergy UK says too many GPs are ignorant about the condition and write off symptoms as nothing serious.
But it can leave children needing treatment for weeping eczema, sores, reflux, vomiting, facial swelling and breathing difficulties.
And even mothers who exclusively breastfeed run the risk of cow's milk proteins being passed to their baby through their own milk.
Cow's milk allergy (CMA) occurs when the body launches an immune response to the protein in cow's milk.
Researchers found that, it takes an average of five months for CMA to be correctly diagnosed.
A spokesman for Allergy UK – which has launched the website isitcowsmilkallergy.co.uk – said: "Undiagnosed CMA leaves infants in distress and puts families through unnecessary anxiety."
More on Parentdish: Does your child have a food allergy or an intolerance?