Alan Johnson has a very nice office. It's on the corner of a building at the end of Whitehall, with gorgeous views over the Palace of Westminster and Parliament Square. It has a little balcony looking down on Whitehall, which the former Labour minister never uses, mainly because he doesn't smoke.

It turns out that the previous occupant of the office used to puff away on the balcony all the time - it was Eric Illsley, the MP whose expenses claims were so venal and vile that he had to resign from the Commons in February 2011 - convenient for Johnson, who voluntarily left the Shadow Cabinet at about the same time, and in doing so forfeited his office over the road near the Commons.

It's nice that someone so clearly unfit for public office was superseded by one of Westminster's all-round good guys, although trying to say that to Alan Johnson is gently dismissed with a friendly, but no-nonsense, wave.

"None of that poncing around," Johnson tells me, as he's describing the move from the heart of Parliament to the outlying building he now works in. He didn't lobby for the office, it just came up. But for 15 years, he always had an office in the actual Palace of Westminster, because for almost the entire time Johnson has been an MP, it hasn't been as a backbencher.

Within a few months of being elected as the Labour MP for Hull in 1997 Johnson was part of the government, rising rapidly through the ranks to be higher education minister, education secretary, health secretary and finally home secretary under Gordon Brown.

Then Labour lost power and during the interregnum at the top of the party - after Brown's resignation and before Ed Miliband took over, he retained his home secretary brief. Then Ed Miliband appointed him shadow chancellor, a brief he'd barely managed to get his head around before revelations emerged that his now ex-wife had been having an affair with his former ministerial bodyguard. He quit the shadow cabinet over the affair, and is not interested in discussing the matter any further.

Strange that this highly capable and genuinely likeable political operator left frontline politics for such reasons, but his current status as a common backbencher is a novelty for him. He's engaged in trying to get a settlement for Hull trawler men, who lost their jobs in the 1970s and 1980s during the Cod Wars with Iceland. He's been fighting to get them compensation, and seems to be close to achieving that.

His other preoccupation is finishing a book - a strange-sounding hybrid between autobiography of his early life growing up in Notting Hill, laced with a history of the west London neighbourhood. There was a bidding war among publishers, apparently.

"What they say is the story of the east end has been done to death, but they liked the absence of the rose tinted glasses I brought to growing up in west London. It’s a kind of theme of mine. I was there, I lived through the 50s, don’t tell me how. The reason my mum ran up such a huge gas and electricity bill that she couldn’t pay and they cut off our heating and our light is because they had to take the meters out, because we were robbed every five minutes. The publishers liked that, they liked the fact that west London had its own coverage."

Is he happy, I ask him?

"Deliriously. QPR beat Liverpool – they came back from two goals down, never happier."

Not bored? "No not at all!" And you believe him, because everyone has always believed Alan Johnson, famous for not mincing his words. It's helped him to get away with things few other politicians can. An interview during his brief spell as shadow chancellor - in which he professed to not know the National Insurance contribution rate for businesses - could have been a disaster. Instead the press acknowledged that he was fairly new in the brief and cut him some slack. You can't imagine Ed Balls ever getting away with that.

You got away with murder in that sense, I tell him.

"I think that comment I made when they asked me what was the first thing I was going to do and I said, 'Read an economics primer', was a throwaway remark. Actually if you back and look at the history of chancellors of the exchequer… Gordon when he came in as Shadow Chancellor didn’t know much about it. That was said to be a big error by lots of people. But, fuck it, just say what you feel at the time."

That "say what you feel" mantra dominates the whole interview. Does he miss being in the Shadow Cabinet? He wishes he'd never joined it.

"I should have done what Jack and Alastair did," he says, referring to Jack Straw and Alastair Darling, both of whom left the Shadow Cabinet and headed for the backbenches after Ed Miliband was elected leader.

"We all stayed in our posts and sort of crystallized ourselves into the posts we held in government, in opposition, to see through that period of electing the leader. I was then Shadow Home Secretary and then Shadow Chancellor – it is a lot of work. Not so much responsibility, not so much fun and not so many staff to help you."

So being in the Shadow Cabinet sucks?

"Well when we came to the end of that period and the new leader was elected, Jack shuffled onto the backbenches and so did Alastair, and really so should I – just in the sense that there were lots of new people coming through and to them a job in the Shadow Cabinet was quite important, it was a step up. To us, it was bloody depressing, it was like a mirror image of where you worked – you were on that side of the Commons looking at that side of the Speaker – and suddenly you were on the other side.

"Every time I heard someone refer to the Home Secretary, I’d listen because I thought they were referring to me – when I went into Home Office Questions you’d think – oh I got to know what questions to answer here, you’d look down and see you didn’t have any questions to answer, just a couple to ask."

We're doing the interview on Thursday, the day after the Budget. I tell him that I thought Ed Miliband's budget response speech had been pretty good.

"Oh yes, he was very good. One of his best performances. He seemed more relaxed. You don’t see Ed relax and smile at the dispatch box. He seemed to be very confident and relaxed, and that came across."

And that's something he needs to do more of?

"He does, as if he is cut out to be at that dispatch box, and aims to be there a long time – the one opposite rather than the one he's at now."

I suggest that Ed Miliband's problem is a lack of consistency - a good performance in the Commons is followed up by some gaffe that immediately undermines him. "I think it took him time to get his office straight, maybe that’s because he didn’t think he was going to win. He hadn’t planned out to any great extent how he was gong to run his office. I think he is settling down now and some of the stuff that are called gaffes – the Hull City stuff – what can you do about that?

"He actually contacted Hull City on the Friday night to say he might not be able to go, but then he recovered. Some of these things, you can have the best people around you – but you’re never gong to stop a Daily Mail front-page story like that."

Johnson comes across as a critical friend of Ed Miliband, someone who probably would prefer a more Blairite streak to run through the Labour opposition, but who accepts that Miliband has a fine balancing act to perform.

"My simple view about leading the Labour Party back to power is if you’re not upsetting people, you’re not doing your job right. You’ve got to be upsetting people."

I ask him which of the government departments he ran was his favourite. "I was at Health for two years, longer than I spent at most departments. It was probably my favourite but Home Office runs it close. Home Office was really interesting. Counter terrorism, immigration. You couldn’t chuck money at it. It was really interesting, much more fascinating than any other department."

So how's Theresa May doing - almost two years into the job, a longer stretch than most Labour Home Secretaries managed?

"She’s doing alright. I fundamentally disagree with the way she dealt with Brodie Clark, and a lot of things. She’s going to have a big problem if they can't get Abu Qatada back to Jordan in the next three months and his bail conditions are lifted. They’ve not got a control order anymore.

"It was stupid, frankly, of Cameron and May to get rid of control orders. And the problems in policing are going to be the next NHS bill. She has done OK. She has kept quiet. Being a woman helps as Cameron hasn't got many women, and certainly not in senior positions. That has helped her. But she's done the job of home secretary better than a lot of her colleagues would have done."

So will the government fail to deport Abu Qatada? "I can’t see it. Even if they get a deal with Jordan that they're trying to get - which is the right thing to do. My criticism is that they haven’t got control orders to fall back on. Even if they have got that agreement, he has all the avenues to appeal on, which he will use. Maybe in six months, but three months will be tough."

It sounds like the sort of thing that keeps home secretaries awake at night. Was he glad he wasn't in charge when the riots took place last August?

"I could have handled these things when Home Secretary. I was Home Secretary when a guy put 30 ounces of semtex in his underpants and tried to blow up a plane – it was linked back to the UK so I was heavily involved with that. That was one Christmas completely gone. If you’re home Secretary, what was it Mandelson’s grandfather Herbert Morrison said? The walls of the Home Office are paved with dynamite."

So a return to Shadow Cabinet is firmly ruled out. How about becoming a Police Commissioner? Johnson doesn't rule that out, but says "I’d never say never, but it doesn’t float my boat."

What advice does he have for Labour police commissioners, given the party was opposed to their creation in the first place?

"As President Nixon said, once the toothpaste is out of the tube, you can’t put it back again. We’re always going to have elected police commissioners now. Can't see any way that we’re going to go back on it. Whoever does it will try their best I hope, to deal with that tension that is obviously going to be there between the chief constable that is responsible for operational decisions and a police commissioner who can hire and fire them.

But can these commissioners avoid being dragged into operational matters? "Can’t see how they won't!" And his counsel to those about to do the job? "They might have some very interesting discussions behind closed doors, but their most important relationship to them will be how they got on with their chief constable and understanding where their remit begins and ends."

Our time is up. "Is that alright for you?" he asks, as a way of signifying this. I have to ask him one last thing before I'm booted out of his luxurious office, though. Is he dating? "Don’t talk about my private life," he says, beaming. Although by all accounts, he is dating and very much in love again.