Jeremy Hunt was asked to admit how bad the situation was for doctors as he faced questions from GPs.
The Health Secretary took two questions from delegates after speaking at the Royal College of GPs annual conference in Liverpool.
He told GPs there was no “silver bullet” to the problems facing them but the Government was “absolutely committed” to increasing funding and capacity.
One delegate told him two primary care workers in a peer support group she was involved in had committed suicide.
Another urged him to acknowledge “how bad the situation is” and asked for the £350 million which it was claimed would be saved after Brexit to be pledged to GPs.
Mr Hunt said: “I did not say when I was here before that we were going to solve these problems overnight.”
He told the conference: “You can make a big difference in your own practices.
“This isn’t going to be something that the Health Secretary solves for you, it’s going to be something that we solve together.”
In his speech, he said: “There isn’t a silver bullet to the problems and challenges facing GPs but we do have a plan and we are implementing that plan.”
There was laughter in the hall as chairman of the RCGP Professor Helen Stokes-Lampard introduced Mr Hunt and asked delegates to give him a “professional and respectful welcome”.
Health Secretary Jeremy Hunt (Neil Hall/PA)
He told delegates: “Too many of the GPs I meet are knackered, they are often feeling at the end of their tether, they feel that they’re on a hamster wheel of 10 minute appointments, 30 to 40 every day, seem never ending.
“They don’t feel able to give the care that they would like to to their patients and increasing numbers of them are choosing to work part-time and at worst to leave the profession.
“We have to think really hard about how to stop that happening if we’re going to use the magic of general practice to do what we need it to do for the NHS.”
During the speech Mr Hunt pledged a package of measures to help GPs.
They included a one-off payment to try to entice 200 GPs who are beginning their careers to work in areas struggling to secure family doctors – such as rural practices and those by the seaside.
From 2018, surgeries in hard to recruit to areas will benefit from the new £4 million scheme.