The double Bafta-winning actress said she was once desperate to join the likes of Dame Maggie Smith and Hugh Bonneville in Julian Fellowes' hit ITV show.
But the Broadchurch star told the Radio Times that donning her outfit for The Suspicions Of Mr Whicher, the TV adaptation of a 19th Century murder case, had put her off doing more period drama.
"I used to want to be in Downton because I had never been in a period drama, but then I did The Suspicions Of Mr Whicher and had to wear one of those frocks and...I didn't feel very comfortable," the 39-year-old star said.
Colman is due to film a second series of Broadchurch, the hit ITV crime drama co-starring David Tennant, in a few months.
But the Peep Show actress said that when she filmed the first series she had to avoid public transport because she was plagued by fans of the show who wanted to know the identity of the killer.
"In the final couple of weeks of Broadchurch I ended up spending a fortune on cabs, because every time I took the train l would see people nudging each other, whispering, 'Ask her, ask her,'" she told the magazine.
"I was getting texts from friends saying, 'Tell me, tell me.' But I couldn't, because I'd signed all these legal documents swearing me to secrecy. I didn't even tell my mum and dad."
The star won two Baftas this year - for Olympics comedy 2012 and for the drama Accused - but she said that her two boys, aged five and seven, did not watch her Bafta triumph on TV because they were tucked up "in bed".
"My mum and dad had bought a bunch of flowers for them to give me the next day, but they clearly had no idea why or what they were for," she added.
Colman has turned down an approach to take part in Dancing On Ice and said that there was "no way" she would join the Strictly Come Dancing line-up.
"I love watching the show and I appreciate the work that goes into it, but I would not be acting; I would be there as a 'celebrity', and it's not my cup of tea. That's crossing a line," she told the magazine.
The actress said that she could understand why her fictional husband (David Morrissey) in her new BBC1 drama, The 7.39, embarks on an affair with a younger woman who he meets on the train to work.
"I spent years commuting into London when I was working as a temp and I hated the monotony of it," she said.
"David Morrissey's character is at a point in his life when everything is the same every day. He's bored, hates his job, but he's forgotten that there's this fantastic love at home and he has to weigh up whether it's worth losing that."
Despite having considered giving up acting a few years ago, Colman has enjoyed huge success in her acting career.
But she said that she would never go down the route of method acting and stay in character after filming because "I'd feel bad pretending my life was anything other than pretty good, so I do the role as well as I can and then I go home, have a cup of tea, see my family and friends and appreciate what I've got".