Pictures and video posted on social media showed a huge plume of black smoke could be seen right across the capital.
Guests, including one still in a bathrobe, were evacuated from the building.
London Ambulance Service and Metropolitan Police also attended the scene. The cause of the blaze is not yet known but police have ruled out terrorism.
No casualties were reported.
Juno Dawson was inside the hotel doing a photoshoot for Attitude magazine when the fire broke out.
She said when the fire alarm went off, the crew assumed it was a test.
“We carried on shooting. Then it started again, and we saw this huge plume of black smoke,” she said.
“We thought ‘shit the hotel is on fire’, got our things and went to the exit.
“The staff was brilliant, there was someone on every corner directing us out.”
Dawson said staff and guests were evacuated within minutes to Hyde Park, and some people looked “very shaken”.
“We we evacuated right past where the fire was. There was ash raining down on Knightsbridge, right opposite Harvey Nichols.”
Dawson said the fire appeared at the very top of the building, where new penthouses are currently being constructed and are covered in scaffolding.
She said her only concern was that it seemed the fire “was well ablaze” by the time the fire alarm in the building went off.
“We were very calm until we walked past the fire, when you could see the flames and you realised it wasn’t someone setting off an alarm in the bathroom but a proper blaze,” she said.
The hotel has only just undergone a £185 million, 18-month refurbishment headed by designer Joyce Wang.
All 181 bedrooms and suites are “now more luxurious and comfortable than ever before” and the building’s “iconic facade was revitalised”, the company said.
The hotel is located between Hyde Park and the Knightsbridge shopping district, which includes Harrods.
Simon Cockshutt was enjoying a glass of wine on the terrace of the Royal Thames Yacht Club next to the hotel when he spotted a thick column of smoke.
He told HuffPost UK: “There was suddenly all this smoke and when we looked up you could see all these people running down the fire escape stairs on the outside of the building.
“The smoke seemed to stop and two or three people went back and then it all started again and then all the ambulances and fire engines arrived.”
His companion Gerry Johnson, said the fire alarm went off in the yacht club and they were asked to evacuate said after the smoke had been billowing for a couple of minutes there was a bang.
Penelope Evans, 50, from Battersea, south London, was shopping in Harrods when she noticed the thick smoke as she was leaving.
“I saw all this black smoke coming from the sky, it was covering everything and the smell was horrendous.
“It was like people were in shock, everyone immediately just started leaving the street and the area.
“There were hundreds of fire engines, ambulances, police racing by and helicopters overhead.
“After Grenfell, I feel like people are a lot more on edge about these things, people just wanted to get out.”
London Fire Brigade said 97 firefighters and 17 engines were being sent to tackle the fire.
The A4 between Sloane Street and Grosvenor Place has been closed.
Transport for London said the blaze was also affecting entry and exit to Knightsbridge station.
It tweeted: “Knightsbridge: currently the Sloane Street entrance at the station is closed. Customers are advised to use the Harrods entrance. This is at the request of the emergency services.”
Mandarin Oriental Hyde Park, London opened its doors in 1902. Built in an Edwardian style, the five-star hotel has 198 rooms and suites, as well as restaurants and bars. It underwent a complete renovation in the late 1990s, and reopened to the public in May 2000.