The British actor announced earlier this year that he was stepping down as the Time Lord, but has confessed that it was a tough decision, reported The Wrap.
Speaking at the Television Critics Association Press Tour in Beverly Hills, he said: "I've pondered it and I've thought I've made a huge mistake. It has transformed my career. It has been wonderful for my life and my family's life."
Matt, 30, went on: "I've done it for four years and we shoot nine to 10 months per year. The show has come to a natural tipping point. I'll miss it. It's a hard choice."
Much of the panel discussion was devoted to the upcoming 50th anniversary episode of the sci-fi show, which is expected to reunite a number of the Doctor's past incarnations.
Matt said he was thrilled to be part of such a "wonderful legacy" and suggested that he would soon be forgotten about when another actor takes over the Tardis.
He said: "The 50th anniversary will honour the ones before me. Then, the show will get bigger and better. People will forget me."
Producer Marcus Wilson was also at the event and confirmed the search for a new Time Lord was "ongoing".
"You just want the best actor for the role. You just kind of cast the net wide and hope you find something brilliant," he said.