The shadow business secretary told Sky interviewer Dermot Murnaghan that he couldn't say if the letter was "patronising" as he hadn't read the letter, adding that he had been invited on to talk about David Cameron's speech today on the economy.
Murnaghan, however, insisted that he should have read the letter, which has been criticised for suggesting Islam is "apart from British society", suggesting the Labour frontbencher could come back on his show in half an hour once he has read the letter.
Umunna said the presenter was being "ridiculous", adding: "Nobody told me that I was going to come onto this programme and be asked to agree if whether I thought the government was patronising Muslim people and Muslim leaders. I'm not just going to speak off piste without actually having read al etter, I don't think you're being terribly fair.
But Murnaghan was not appeased, concluding: "So you're not going to speak until you get the party line right? OK, so we'll have to end it there."
David Cameron has defended Pickles' letter, which told more than 1,000 Islamic leaders they "had more work to do" to root our extremists, as "reasonable, sensible and moderate".
He added: "Anyone, frankly, reading this letter, who has a problem with it, I think really has a problem."
This will not please The Muslim Council of Britain (MCB), which said the letter appeared to re-enforce claims by the far right about Islam's place in Britain.
MCB deputy secretary-general Harun Khan said: "We will be writing to Eric Pickles to ask that he clarifies his request to Muslims to 'explain and demonstrate how faith in Islam can be part of British identity'.
"Is Mr Pickles seriously suggesting, as do members of the far right, that Muslims and Islam are inherently apart from British society?"
Answering questions after a speech in Ipswich on Monday, Cameron said: "I think it is absolutely right to write this letter, to say that we all have a responsibility to fight extremism.
"Anyone who reads this letter - and I've read the letter - will see that what he is saying is that British Muslims make a great contribution to our country, that what is happening in terms of extremist terror has nothing to do with the true religion of Islam.
"It's being perverted by a minority who have been radicalised. But everyone needs to help with dealing with this problem of radicalisation."
He added: "I think it is the most reasonable, sensible, moderate letter that Eric could possibly have written.
"Frankly, all of us have a responsibility to try to confront this radicalisation and make sure that we stop young people being drawn into this poisonous fanatical death cult that a very small minority of people have created."
In the letter sent to more than 1,000 Islamic leaders, Pickles and communities minister Lord Ahmad stressed that they were "proud" of the way Muslims in Britain had responded to the Paris terror attacks but added that there was "more work to do".
"You, as faith leaders, are in a unique position in our society. You have a precious opportunity, and an important responsibility, in explaining and demonstrating how faith in Islam can be part of British identity," they wrote.
"We believe together we have an opportunity to demonstrate the true nature of British Islam today.
"There is a need to lay out more clearly than ever before what being a British Muslim means today: proud of your faith and proud of your country. We know that acts of extremism are not representative of Islam, but we need to show what is."