The Strictly Come Dancing judge branded the media mogul's actions "spiteful" after Cowell said he brought Dixon to Britain's Got Talent to hurt his dance rival.
Goodman - who is an acquaintance of the talent show boss - told Radio 5 Live that he was "disappointed" in Cowell if that was his motivation.
And he accused him of spoiling the pleasure of millions of viewers.
Dixon walked out on Strictly earlier this month in a shock move after three years with the series.
Speaking about his signing, Cowell said: "My decision to poach her was 70-30 on the fact she was on Strictly and the fact I liked her.
"With us and the BBC, I laugh about it, but there is unbelievable rivalry now. They want to be number one, I want to be number one. We do what it takes."
But Goodman hit back, telling radio host Richard Bacon: "I would never knock the opposition and I think it is spiteful and mean if that is the case that Simon said 'I just want to be spiteful towards Strictly'.
"Because why would you want to do that? Why would you want to spoil millions of people's pleasures? It's not the right way to behave, so if that is the case - shame on you, Simon."
Goodman said he would welcome a return to Strictly for choreographer Arlene Phillips. The 68-year-old was axed from the judging panel and replaced by Dixon, 35 years her junior, which led to accusations of BBC ageism.