The President confirmed the news in a tweet announcing Mike Pompeo, the current Director of the CIA, will take his place.
Hours later, Tillerson held a news conference where he notably did not mention Trump by name when name-checking colleagues he had worked with, as he made a pointed warning about Russia aggression.
He revealed he received a phone call from Trump - calling him around noon from Air Force One - hours after his removal on social media.
A briefing war played out before Tillerson, whose voice quivered at times, spoke to reporters.
A senior White House official said Trump asked Tillerson to step down on Friday but did not want to make it public while he was on a trip to Africa.
But a State Department spokesman told CNN that Tillerson found out about his firing when Trump announced it on Twitter on Tuesday.
Reporters travelling back with the Secretary say he had no idea the move was coming.
Speaking to reporters outside the White House shortly after the announcement was made, Trump said he made the decision “by myself” and that Tillerson will be “much happier now” .
He added: “Rex and I have been talking about this for a long time.
“We actually got along... quite well but we disagreed on things.”
But the State Department made clear in a statement that Tillerson did not wish to step down.
“The secretary had every intention of staying because of the critical progress made in national security. He will miss his colleagues at the Department of State and the foreign ministers he has worked with throughout the world.
“The secretary did not speak to the President and is unaware of the reason, but he is grateful for the opportunity to serve, and still believes strongly that public service is a noble calling.
“We wish Secretary Designate Pompeo well.”
Reports of such a move have been circulating since as far back as November of last year.
The resignation represents the biggest shakeup of the Cabinet so far, adding to a list unprecedented in presidential history.
And in a further sign of the ongoing chaos in the White House, John McEntee, the President’s personal assistant, was escorted off the premises for an unspecified security issue.
Trump and Tillerson have had a rocky relationship with the President often undermining much of his work as Secretary of State.
Just yesterday both appeared at odds over the response to the poisoning of the former Russian spy and his daughter in Salisbury.
During a press briefing on Monday afternoon, Trump’s spokeswoman Sarah Huckabee Sanders refused three times lay the blame on Vladimir Putin’s Government.
Then, just hours later, Tillerson said the attack “clearly came from Russia”.
Tillerson also appeared out of the loop last week when Trump announced he would meet with North Korea’s leader and become the first sitting US president to do so.
The spat has also become embarrassingly public on Twitter at various points.
Last October NBC News reported that Tillerson called Trump “a moron” during a meeting with top officials over the summer.
Tillerson said his tenure ends on March 31 but he would delegate his responsibilities to John Sullivan, deputy secretary of state, at the end of Tuesday.
“What is most important is to ensure an orderly and smooth transition during a time that the country continues to face significant policy and national security challenges,” Tillerson told reporters.
Tillerson had developed a strong relationship with Secretary of Defense Jim Mattis, and the two were seen as a moderating influence on some of Trump’s policies.
“I’m told for the first time in most people’s memory the Department of State and the Department of Defense have a close working relationship where we all agree the US leadership starts with diplomacy,” he said.
Pompeo’s replacement at the CIA will be Gina Haspel, notable for her prominent role in overseeing the torture of detainees in Thailand in 2002 in the early fight against al Qaeda.
Working as a clandestine officer in Thailand in 2002, Haspel reportedly was involved in the interrogations of two suspects, Abu Zubaydah and Abd al-Rahim al-Nashiri, several news outlets reported last year.
The methods used against the men included waterboarding Zubaydah 83 times in one month, to the point where doctors once had to revive him, and ramming his head into walls. He lost sight in one eye.
Haspel’s likely promotion reflects Pompeo’s and Trump’s sympathetic approach to torture. Trump has said he wants to bring back waterboarding. Pompeo said he would consider reinstating it, although he couldn’t imagine that Trump would ask him to.
Tillerson’s days numbered from the start
A government outsider lacking diplomatic experience, Tillerson arrived at the White House after decades at Exxon Mobile, rising from engineer in 1975 to chairman and CEO in 2006, overseeing vast expansion of the company’s operations worldwide.
Trump picked Tillerson for the State Department job after a lengthy public vetting of prominent candidates, including former New York Mayor Rudy Giuliani and former Republican presidential candidate Mitt Romney.
Tillerson faced opposition during his Senate confirmation process, with lawmakers expressing concerns over his lack of experience in government and diplomacy, and strong ties with Russian leader Vladimir Putin from years of business in the country.
Even Tillerson himself had been reluctant to accept the position. “I didn’t want this job. I didn’t seek this job,” he admitted last March.
“My wife told me I’m supposed to do this,” he responded when an aide asked him why he accepted the position anyway.
During his tenure, the department faced drastic budget cuts and a crucial leadership vacuum. Amid massive international challenges like North Korea’s nuclear arms race and the war in Syria, Trump proposed cutting the State Department budget by nearly one-third. Dozens of vacancies remained unfilled, including top deputies, because of White House refusal to accept Tillerson’s picks.
Tillerson and the White House also clashed on policy and messaging.
As a diplomatic crisis in Qatar unfolded, Trump directly undermined Tillerson’s calls for calm, expressing support for Saudi Arabia’s economic and diplomatic blockade of the Gulf state, and accusing Qatar of funding terrorism.
In an August interview, Tillerson declined to defend Trump’s support for “America’s values,” telling Fox News that “the president speaks for himself.” The questioning arose during a conversation about Trump’s much criticised response to violence at a white supremacist rally in Charlottesville, Virginia, earlier that month, when counter-protesters clashed with hate groups protesting the planned removal of a statue honouring Confederate General Robert E. Lee.
From the White House, the president’s senior adviser and son-in-law, Jared Kushner, often ran a parallel track of diplomacy ― traveling to Israel and the Palestinian territories to explore ways to revive peace talks, and interacting with Mexican President Enrique Peña Nieto.
Tillerson is just the latest to step down in the Trump administration. In September, Health and Human Services Secretary Tom Price was forced to resign, after he used taxpayer-funded private jets on numerous occasions.
National security adviser Michael Flynn was forced to step down in February following revelations that he discussed US sanctions against Russia with that country’s ambassador before Trump’s inauguration.
Trump fired FBI Director James Comey last May, who had been leading an investigation into whether Trump’s campaign colluded with Russia to influence the presidential election.
Mike Dubke, White House communications director, resigned later in May after months of chaos in the press shop. Press secretary Sean Spicer quit in July, after the president appointed Anthony Scaramucci as the new communications director. Scaramucci resigned after just 10 days, amid a spate of departures that included White House Chief of Staff Reince Priebus and, less than a month later, chief strategist Steve Bannon.