The Take That singers now face tax bills of around £20 million.
The Times reports that Judge Colin Bishop told the court that “Icebreaker is, and was known and understood by all concerned to be, a tax avoidance scheme.”
Now Labour’s Margaret Hodge, chair of the Commons Public Accounts Committee, said Barlow “might want to show a bit of contrition by giving back his OBE”.
But David Cameron has said he does not think the singer should be forced to give back his OBE, despite publicly attacking the comedian Jimmy Carr last year for his involvement with a tax avoidance scheme.
The Prime Minister, who appeared with the Tory-supporting Take That star during the 2010 General Election campaign, said: "Gary Barlow has done a huge amount for charity, raising a huge amount for Children in Need."
Mr Cameron went on to tell Sky News that the singer was appointed is OBE for his charity work and the tax avoidance issue shouldn't affect that.
Gary has yet to comment on the reports that he and his bandmates will have to repay millions but at the time of investing, lawyers for Gary, Howard and Mark confirmed they were involved in the partnerships but insisted the trio believed they were “legitimate enterprises”.
Gary Barlow
(01 of11)
Open Image Modal
Take That lead singer Gary Barlow at the Smash Hits Awards in London, where they won seven categories, including Best Group and Best Single Video.
(02 of11)
Open Image Modal
Take That perform on stage at the Brit Awards. L-R; Jason Orange, Gary Barlow, Howard Donald.
(03 of11)
Open Image Modal
Gary Barlow London, July 1999 Pic Credit: WENN/ Paul Doyle
(04 of11)
Open Image Modal
Gary Barlow Take That performing at the NEC Birmingham Birmingham, England - 26.04.06 Credit: (Mandatory): Danny Clifford/Hottwire.net/WENN.com
(05 of11)
Open Image Modal
Gary Barlow signs copies of his book 'My Take' at Waterstones London, England - 09.10.06 Credit: Lia Toby / WENN
(06 of11)
Open Image Modal
Gary Barlow and Ronan Keating perform a duet together of the song 'Harvest For The World' on stage at the Concert of Hope in aid of the Diana Princess of Wales Memorial Fund.
(07 of11)
Open Image Modal
Take That Performing live at a special showcase for their fans to launch their debut album "Take That & Party" The Hippodrome London Credit: (Mandatory): George Chin / WENN.com
(08 of11)
Open Image Modal
Jason Orange, Mark Owen, Gary Barlow and Howard Donald of Take That pose at a signing session for their new Album 'Beautiful World' at HMV on November 27, 2006 in London, England. (Photo by Jo Hale/Getty Images)
(09 of11)
Open Image Modal
Jason Orange, Gary Barlow, Mark Owen and Howard Donald of Take That performing in concert at the O2 Arena in Greenwich, London. *** Local Caption *** Jason Orange;Howard Donald;Gary Barlow;Mark Owen
(10 of11)
Open Image Modal
Pop group Take That backstage. (l-r) Robbie Williams, Howard Donald, Mark Owen, Jason Orange and Gary Barlow
(11 of11)
Open Image Modal
Singers Gary Barlow and Cheryl Cole perform on stage during the Diamond Jubilee concert at Buckingham Palace on June 4, 2012 in London, England. For only the second time in its history the UK celebrates the Diamond Jubilee of a monarch. Her Majesty Queen Elizabeth II celebrates the 60th anniversary of her ascension to the throne. Thousands of well-wishers from around the world have flocked to London to witness the spectacle of the weekend's celebrations. (Photo by Dan Kitwood/Getty Images)