Viewers saw Matthew Crawley killed off in the Christmas special, just after he and wife Lady Mary had welcomed their first child.
Fellowes admitted to The Hollywood Reporter that after Matthew and Mary's long love affair, the only way to write him out was to kill him off. But as Jessica Brown Findlay's character Lady Sybil had only just been killed off a few episodes earlier, they did not want to dwell on the death.
He said: "We knew Jessica was leaving from quite a long time before, and so we completely decided calmly to kill her off in the fifth episode, so that would give people three episodes to recover, and then we'd have the Christmas special and everyone could move forward. So we had a whole episode of her dying.
"But having done that, when the news came through that Dan was also leaving we couldn't very well do it again, so that was a bit testing."
Stevens revealed he was leaving just as series three started filming, and Fellowes admitted at first he did not know how they were going to write him out of the award-winning period drama.
But executive producer Gareth Neame said: "We were all agreed that the only way we could break Mary and Matthew apart was for one to die. There really was no alternative."
Fellowes added: "We couldn't have three years of them being in love and suddenly him taking against her and nipping off to Japan. That wasn't a workable scenario."
Stevens previously apologised for his character's untimely death, but insisted he had no say in the matter. He told the Radio Times: "We didn't see that script until the very last minute, so we didn't know exactly how they were going to do it. I didn't have any say in the manner in which he went. Ultimately, it was in the hands of Julian and the producers."
"It was right that he didn't run off and have an affair with somebody. I don't think that would have been right for Matthew as a character."
'Downton' will return for an eight-part fourth series and a Christmas special in the autumn on ITV. Check out the pictures below...