The Liberal Democrat beat the incumbent by several thousand votes overnight despite the independent MP previously having boasted of a 23,000 majority.
Responding to Olney’s surprise victory, Ben Goldsmith, Zac Goldsmith’s younger brother, wrote in a now deleted Tweet: “Even by Lib Dem standards Sarah Olney really is unimaginably drab.”
Commenters on Twitter called the response “bitter” and said it seemed the only thing the Goldsmith family knew how to do was “lose disgracefully”.
Ben’s tweet echoed the sentiment expressed by his sister Jemima Khan who chose to attack Goldmith’s London Mayoral election campaign after he lost to Sadiq Khan in May, rather than extending her congratulations to the victor.
The comment came just as Khan was declared the winner, in what had been an increasingly poisonous race.
Towards the end of the campaign Tory attacks on Khan increasingly tried to link him to Islamic extremism and terrorism.
Before the Richmond Park result was announced on Friday, Ben Goldsmith was confident his brother would be victorious.
But in the end Goldsmith, who quit the Conservative Party over the Heathrow runway decision, only received 18,638 votes.
Olney bagged 20,510.
The Lib Dems focussed their campaign on Brexit in the staunchly Remain west London seat.
Olney, who took 49.7% of the vote, said dissatisfaction with the way Brexit was being handled was the key factor in her victory.
She said the result was a rejection of the “politics of anger and division”, telling supporters: “We will defend the Britain we love. We will stand up for the open tolerant, united Britain we believe in.
“The people of Richmond Park and North Kingston have sent a shockwave though this Conservative Brexit government, and our message is clear: we do not want a hard Brexit. We do not want to be pulled out of the single market, and we will not let intolerance, division and fear win.”
Labour’s Christian Wolmar trailed a distant third with 1,515 votes, after the Greens backed the Lib Dem candidate, while Ukip stood aside to give Goldsmith a clear run and the Conservatives did not field a standard bearer.
A Conservative Party spokesman said the result would make no difference to Brexit plans.