According to the paper, it was felt he couldn't match the chemistry that Olly shared with Caroline when they presented the show together.
"Matt's a nice guy and did a decent job but he didn't prove a huge hit with fans or producers," a source said.
"It was very hard for him to follow Olly Murs' amazing partnership with Caroline and it showed."
As the story circulated on Sunday, Matt cryptically tweeted: "I really don't like Sundays."
However, his TV partnership with Caroline looks set to continue in another guise, as The Sun reported last month that the pair have landed their own ITV2 show.