The former chancellor, who is now editor of the Evening Standard newspaper, said was “not keen” on holding a vote on the UK’s membership of the EU.
Osborne, who campaigned for a Remain vote, told BBC Radio Four’s Today programme: “I was not keen on having a referendum in the first place but I chose to be part of the collective decision-making of the government.
“I certainly did not walk out of the government at the time. That is not the way I tried to behave as chancellor. I was not trying to be the difficult next door neighbour.
“I was trying to be the person who, with David Cameron, my friend, made things work for the country.”
Osborne said, as the vote approached, he had become increasingly pessimistic about the chances of a Remain victory.
“I was always quite gloomy about the referendum and its outcome,” he said.
“I went round the country and I kept hearing from people I would assume wanted to stay in the EU that they wanted to leave so I was pretty pessimistic.”
Osborne acknowledged his strong support for Remain meant he was a divisive figure within the Conservative Party, but said he had not ruled out a return to politics at some point in the future, the Press Association reported.
“I don’t rule out going back into politics but it is certainly not plan A,” he said.
He said he remained friends with environment secretary Michael Gove, despite having been on opposite sides in the referendum campaign.
“I am still friends with Michael Gove. Lots of my other friends can’t understand it but I like Michael,” he said.
Osborne said, while he could not see the decision to leave the EU being stopped, there was still scope to debate the terms on which Britain withdraws.
“I don’t accept that just because we are leaving the EU we have to, for example, leave the customs union or radically clamp down on immigration,” he said.