This week, the 40-year-old announced after his concert in Vienna that he was going to be a father for the second time.
And in an interview with The Sun, he said he needed to take a step back from re-joining Take That. Robbie said: "Logistically I can't do it. I have the tour and I'm going to be a dad again."
Gary Barlow, and bandmates Mark Owen, Howard Donald and Jason Orange, have been in the studio producing Take That's upcoming album while Robbie is on a 46-date Swings Both Ways tour.
However, Robbie stressed it was not the end of his time with the the band.
He said: "It doesn't mean to say I won't join in along the way. Whatever happens, I am absolutely going to be in Take That - I love being in Take That."
But he added: "After the tour wraps, I'm going to go away for a year or two."
Robbie's wife, Ayda Field is around three months' pregnant and due to give birth in November.
Robbie said: "We don't know if it's a boy or a girl but we're going to find out." But as he is already dad to 20-month-old daughter Theodora Rose - known as Teddy - Robbie said he really would love to have a son.
He said: "If we have a girl, it'll be totally cool but I would quite like a little boy."
The baby's sex will be revealed at the next ultrasound scan, but Robbie and Ayda plan to find out the same way they had arranged it with their first child.
He said: "With Teddy, the doctor put the sex into an envelope. We went to dinner and the plan was to open it at the end of dinner... but she forgot the envelope.
"I think we're going to do the same thing - but remember the envelope this time. I don't think we can do it any other way.
"Last time I said that as long as it was healthy, I didn't mind. Then we found out we were having a girl and it was a really strange emotion. I was like, 'What am I going to do with a girl?'"
Robbie is on the road with Ayda and Teddy, plus his dad Pete Conway and his wife, as well as the mother-in-law.
Robbie said: "It's a very loving, very safe environment. Our biggest concern these days is that we don't travel through Teddy's nap time.
"We don't get time to worry about us or the show. We've got it all going on backstage. Ayda is the happiest and centred and then she's completely not for a few hours, the whole rollercoaster of emotions... "
Robbie's Swings Both Ways tour arrives in the UK on June 13 in Belfast.