David Davis has spent up to an hour in Brexit talks with European Commission chief negotiator Michel Barnier before officials got on with the "serious business" of detailed negotiations, a Government source said.
The Brexit Secretary's meeting during a flying visit to Brussels included 10-15 minutes alone and away from officials with his opposite number Mr Barnier, in discussions described as "very friendly" and "positive".
Shortly after declaring it was time to "get down to work" in a media event alongside Mr Barnier, Mr Davis returned to London.
The source dismissed suggestions Mr Davis had to leave talks early to get back to the House of Commons in time to vote against a Labour motion to vastly increase the amount of debating time for backbenchers' legislation.
But the speculation came with the minority Government mired in difficulties, with Theresa May set to order senior ministers to keep Cabinet discussions private after a series of leaks targeting Chancellor Philip Hammond.
Labour said Mr Davis's exit from Brussels after a few hours, amid the storm of leaks at home, showed the Government is in "disarray", while the Liberal Democrats mocked the Brexit Secretary for "skulking back to the UK after just half a day".
Mr Davis had also spent time speaking to some of the 98 UK officials carrying out the detailed talks, and a Government source insisted: "He didn't go back for anything specific, just that the timetable for today was always set out this way, that he would go back this morning, they had already agreed that."
During their meeting, Mr Davis and Mr Barnier talked about tasking their teams for talks, and said they were looking forward to getting into the substance of the issues, the source said.
Mr Davis is expected to return for further talks with Mr Barnier on Thursday and a potential press conference to wrap up the second formal round of negotiations.
The four day talks got under way with a plenary session followed by working groups on the issues of citizens' rights, borders, the UK's financial settlement and separation issues, including the UK's position in relation to the European civil nuclear regulator, Euratom, the source of Tory disquiet at home.
The mood among the 98 British officials in Brussels was said to be "determined" and a source said they "covered quite a lot of ground" in the talks, conducted in English and French.
A spokesman for the Department for Exiting the European Union said: "As the Secretary of State has said, it's time to get down to business.
"Both sides have today got round the table and started the serious business of working through our positions in a number of areas.
"We recognise that this will be a complicated and technical process and we look forward to coming back tomorrow to make progress on the work we have begun today."
The negotiations were overshadowed by an explosive Cabinet row in Britain.
The Prime Minister is set to use Tuesday's regular Cabinet meeting to "remind" ministers they should maintain silence about the content of meetings and focus on their job of delivering for the public.
It comes after a series of newspaper headlines about Mr Hammond's comments at last week's Cabinet, culminating in a Daily Telegraph front page story quoting an unnamed minister accusing the Chancellor of trying to "f*** up" Brexit.
Shadow Brexit secretary Sir Keir Starmer said: "Since the election the Government has been in disarray. There is no agreed Cabinet position on vital Brexit issues, the negotiating team is not prepared and the Prime Minister has lost her authority.
"Meanwhile the clock is ticking and the risks are increasing day by day.
"David Davis can hardly say this is the time to get down to business and then spend only a few minutes in Brussels before heading back to Whitehall.
"We need a fresh approach and to see real progress in negotiations. That means engaging with the substance of talks. And it means resolving vital issues such as citizens' rights that have already dragged on for too long."
As talks opened in Brussels, Mr Davis and his team were photographed sitting without any papers opposite EU negotiators with stacks of papers on the table in front of them.
Liberal Democrat Brexit spokesman Tom Brake said: "This is a Government with no papers, no plan and no time for the most important negotiations of a lifetime.
"They are meant to be negotiating Brexit, but they can't even negotiate among themselves."