The double Bafta-winning actress said she was once desperate to join the likes of Dame Maggie Smith and Hugh Bonneville in Julian Fellowes' hit ITV show.
But the Broadchurch star told the Radio Times that donning her outfit for The Suspicions Of Mr Whicher, the TV adaptation of a 19th Century murder case, had put her off doing more period drama.
"I used to want to be in Downton because I had never been in a period drama, but then I did The Suspicions Of Mr Whicher and had to wear one of those frocks and...I didn't feel very comfortable," the 39-year-old star said.
Colman is due to film a second series of Broadchurch, the hit ITV crime drama co-starring David Tennant, in a few months.
But the Peep Show actress said that when she filmed the first series she had to avoid public transport because she was plagued by fans of the show who wanted to know the identity of the killer.
"In the final couple of weeks of Broadchurch I ended up spending a fortune on cabs, because every time I took the train l would see people nudging each other, whispering, 'Ask her, ask her,'" she told the magazine.
"I was getting texts from friends saying, 'Tell me, tell me.' But I couldn't, because I'd signed all these legal documents swearing me to secrecy. I didn't even tell my mum and dad."
The star won two Baftas this year - for Olympics comedy 2012 and for the drama Accused - but she said that her two boys, aged five and seven, did not watch her Bafta triumph on TV because they were tucked up "in bed".
"My mum and dad had bought a bunch of flowers for them to give me the next day, but they clearly had no idea why or what they were for," she added.
Colman has turned down an approach to take part in Dancing On Ice and said that there was "no way" she would join the Strictly Come Dancing line-up.
"I love watching the show and I appreciate the work that goes into it, but I would not be acting; I would be there as a 'celebrity', and it's not my cup of tea. That's crossing a line," she told the magazine.
The actress said that she could understand why her fictional husband (David Morrissey) in her new BBC1 drama, The 7.39, embarks on an affair with a younger woman who he meets on the train to work.
"I spent years commuting into London when I was working as a temp and I hated the monotony of it," she said.
"David Morrissey's character is at a point in his life when everything is the same every day. He's bored, hates his job, but he's forgotten that there's this fantastic love at home and he has to weigh up whether it's worth losing that."
Despite having considered giving up acting a few years ago, Colman has enjoyed huge success in her acting career.
But she said that she would never go down the route of method acting and stay in character after filming because "I'd feel bad pretending my life was anything other than pretty good, so I do the role as well as I can and then I go home, have a cup of tea, see my family and friends and appreciate what I've got".
1. She lives in South London with her actor-turned-writer husband Ed Sinclair and their two children Finn, seven and five-year-old Hal
2. She played Bev in the Bev and Kev AA car insurance ads
3. She was described by her 'The Iron Lady' co-star Meryl Streep as 'divinely gifted' during her BAFTA acceptance speech for Best Actress following her portrayal of Margaret Thatcher's daughter, Carol.
4. She originally trained to be a teacher but realised it wasn't for her. “I would have been the sort of primary teacher who never noticed the struggling kids because I would have been struggling so much myself,” she later said.
5. She met her 'Peep Show' co-stars David Mitchell and Robert Webb when they were all students at Cambridge University
6. She got her kit off in the Brit flick, 'Confetti', opposite her 'Peep Show' co-star Robert Webb, something she later described as 'the worst experience of my life'
7. Her performance in the 2011 film Tyrannosaur received critical acclaim and bagged her several high profile awards including the British Independent Film Award for Best Actress, the Empire Award for Best Actress, Chlotrudis Award for Best Actress, the London Film Critics Circle Award for British Actress of the Year and the Sundance Film Festival World Cinema Special Jury Prize for Breakout Performances. Phew.
8. Her TV roles have included *takes deep breath* Doctor Who, Peep Show, Holby City, The Office, Green Wing, Skins, Beautiful People, Midsomer Murders, Rev, Twenty Twelve, Accused and of course, Broadchurch.
9. Olivia almost didn't take on the role of DS Miller in 'Broadchurch' because it involved being away from her two young sons whilst she filmed the West Country. “I don’t take jobs away from London because I don’t like being away from the boys," she said. "This is the longest I’ve spent away from them ever.”