The 'King's Speech' star was meant to be providing the voice of the bear, but has told Empire magazine of his tough decision to quit the role.
He told the publication: 'It’s been bittersweet to see this delightful creature take shape and come to the sad realisation that he simply doesn't have my voice.
'After a period of denial we've chosen conscious uncoupling. I've had the joy of seeing most of the film and it's going to be quite wonderful.
'I still feel rather protective of this bear and I'm pestering them all with suggestions for finding a voice worthy of him.'
'Paddington', based on the books and TV animation of the same name, would have been the third film in a row uniting Colin Firth with Nicole Kidman.
The Oscar-winning pair co-starred in the recent 'The Railway Man', telling the story of traumatised Eric Lomax and his wife.
And they will both be appearing in the forthcoming thriller, 'Before I Go To Sleep', in which they play another married couple, with Nicole's character suffering from autograde amnesia, who forgets who she is each day.
Other actors remaining alongside Nicole for 'Paddington' include Julie Walters, Peter Capaldi, Sally Hawkins and Matt Lucas. The film is set for release in November.
Who would you like to see in the role of Paddington? Share your ideas below...