The Him & Her star told The Independent he was relieved not to get the part in the BBC sci-fi series.
He said: "It would have terrified me. I don't know what I would have done with all the attention."
Russell, 31, had appeared in the 2007 Doctor Who Christmas Special and was thrown into the series spotlight when producer Russell T Davies blogged that he thought the actor would be an ideal replacement for David Tennant.
The Being Human star told The Independent: "Russell texted me to say, 'Look, I've said this quote about how you should be the next Doctor Who, so if it goes a bit crazy...' And I was, 'Oh, that's fine...', and then suddenly it just went everywhere: people messaging me from my childhood and lots of remarks like, 'His ears are too big for the Tardis, they'll be flapping about.'
"People were starting to get really personal."
After Russell tested to be the 11th Doctor it was Matt who eventually got the part.
Russell, currently appearing in ITV sitcom The Job Lot, would have been the first openly gay actor to play the Doctor, but said he is happy to have been cast as only straight characters so far.
Russell, said: "For me to play gay it has to be something special, because it might actually be more of a risk. So I'm waiting for that role - I want it to be something that moves things forward."