Culture Secretary John Whittingdale is facing calls to withdraw from the regulation of the press following the disclosure that he had a relationship with a prostitute.
Mr Whittingdale, who is divorced, said he had been unaware of the woman's occupation and had broken off the relationship when he discovered she was trying to sell the story to the newspapers.
However campaigners for tighter press regulation have warned his position was compromised after it was reported that a number of newspapers had investigated the claims but decided not to run the story.
Labour shadow cabinet minister Chris Bryant, who was previously shadow culture secretary, told the BBC: "It seems the press were quite deliberately holding a sword of Damocles over John Whittingdale.
"He has a perfect right to a private life but as soon as he knew this he should have withdrawn from all regulation of the press."
The relationship occurred before Mr Whittingdale became a Cabinet minister following the 2015 general election although he was chairman of the influential Commons Culture, Media and Sport Committee at the time.
He said the relationship had not affected any of the decisions he had made in office.
"Between August 2013 and February 2014, I had a relationship with someone who I first met through Match.com. She was a similar age and lived close to me," he said.
"At no time did she give me any indication of her real occupation and I only discovered this when I was made aware that someone was trying to sell a story about me to tabloid newspapers. As soon as I discovered, I ended the relationship.
"This is an old story which was a bit embarrassing at the time. The events occurred long before I took up my present position and it has never had any influence on the decisions I have made as Culture Secretary."
BBC2' s Newsnight reported that four newspapers – The People, The Mail on Sunday, The Sun and The Independent on Sunday – had investigated the claims but had concluded it was not a public interest story.
However campaigners for tighter press regulation accused the papers of hypocrisy pointing out that as chairman of the Culture Committee, Mr Whittingdale had opposed statutory regulation.
Brian Cathcart of the Hacked Off campaign group said that since becoming Culture Secretary with responsibility for the media, he had taken a number of decisions which had been welcomed by the press.
"The public cannot have faith in his judgment, in his independence in making decisions about the media," he told Newsnight.
"It is not a story about John Whittingdale's private life. It is a story about why the press didn't cover this.
"To suggest in the very week we have newspapers baying for the right to cover a story about a celebrity's private life which a judge has told them they have no right to cover, they would be too scrupulous, too high-minded to report a story about a Cabinet minister which any judge in the country would tell them they have a right to cover is just absurd."
However the media commentator and former newspaper editor Roy Greenslade said that the papers would have been wary about covering such a story in the aftermath of the Leveson report on press standards.
"They would all be very careful about whether or not they had a public interest justification," he told the programme.
"They would have all taken separate legal advice, they would have all looked at their code of practice. I think it is a bit much to castigate the newspapers for doing the right thing for once."