The judge branded the 'Impossible' singer a "competition winner who got lucky" who "needs to be more grateful", following a controversial time for James.
Louis told OK magazine: "He's a competition winner that got lucky. He came from a little bedsit, he should be a lot more grateful and a lot more respectful to everybody. Don't get me going.
"Nicole [Scherzinger] worked so hard with him last year - she got him into the final. He doesn't look comfortable in his own skin. He has talent but he's not Bob Dylan."
Referring to James' recent spat with fellow 'X Factor' stars One Direction, Louis added: "You can't be slagging off One Direction, you can't slag off success. I think they're good role models and show the pop business is alive and well."
"It's been an amazing year for me. There's been some amazing highs but also some lows," he explained to host Dermot O'Leary.
"I've made some very silly mistakes. I want to thank 'The X Factor' for giving me this opportunity.
"I want to say sorry for abusing my position as an 'X Factor' winner because I owe everything to this thing."
He was recently given time off from his management following the outbursts, which they said was down to "exhaustion", but has since returned to the stage.
They have also taken control of the star's Twitter account - the place where James started most of his feuds.