Some 54% said their sex lives had suffered compared to 30% of women of all ages living with the disease.
The poll of 532 women for Macmillan Cancer Support also found 16% of women had sex less often because it was too physically painful.
Meanwhile, 16% said it was due to their partner wanting to have sex less often, 45% blamed feeling body conscious and 69% admitted to having a lower sex drive.
Almost a third (30%) of women who were diagnosed 10 years ago or more said they had sex less often as a result of their cancer.
There are 570,000 women in the UK living with breast cancer.
Story continues below the slideshow:
Jennifer Gorrie, cancer information nurse specialist at Macmillan Cancer Support, said: "Surgical wounds or skin damage from radiotherapy can cause some breast cancer patients' chests to be tender for a time, making some sexual positions painful.
"Sexual intercourse can be uncomfortable too due to chemotherapy or hormonal treatments like tamoxifen causing menopausal symptoms, such as hot flushes and vaginal dryness.
"There are practical ways breast cancer patients can ease the discomfort so they can have a healthy sex life.
"They should speak to their GP or cancer specialist for help or get advice from Macmillan Cancer Support.
"It is vital that these women are given the right support at the right time to ensure they're not dealing with this alone."
Dr Daria Bonanno, consultant clinical psychologist at the charity, said: "Many breast cancer survivors find their sex lives can fall apart.
"As well as dealing with pain, they may find scarring from surgery or radiotherapy has affected their body confidence or treatment has dampened their libido.
"The impact of a waning sex life can be tough, chipping away at a patient's self-esteem and, in extreme cases, leading to depression or contributing to a relationship breakdown."