In an interview with the Radio Times magazine, the 'Bridesmaid' star reportedly said: "Generally my favourites go out in the first few weeks and, of course, because the viewing public is so racist, all the black, strong singers go out in round five."
Despite this, the Irish actor claims he is a "big X Factor fan".
His comments come as part of an ongoing debate about online abuse black contestants on the show receive.
Most recently, 2013 finalist Hannah Barrett said she had been "disappointed" by the racist abuse she received while she was part of Simon Cowell's singing contest.
She told Reveal magazine: "People say I'm too dark to be a singer and there have been lots of references to how dark my skin is".
However, an 'X Factor' source told the Daily Mail: "The X Factor has a long standing record of celebrating a diverse range of artists including two former winners, Leona Lewis and Alexandra Burke, and numerous others who have reached the final stages of the competition including this year’s semi-finalists Rough Copy, along with JLS, Marcus Collins and Jahmene Douglas in previous series."