In feisty exchanges during the live Sky News and Channel 4 televised leaders' interviews, Miliband was forced to defend his decision to challenge his brother David for the Labour leadership. He also rejected the suggestion he was too much of a "geek" to lead the country.
Paxman, who earlier in the evening had come out swinging against David Cameron, told Miliband: "You know what people say about you, because it's hurtful, but you can't be immune to it.
"A bloke on the tube said to me last week: 'Ed Miliband goes into a room with Vladimir Putin, the door is closed, two minutes later the door is open again and Putin is standing their smiling and Ed Miliband is all over the floor in pieces'."
Paxman added: "People think you're just not tough enough."
Miliband replied: "Am I tough enough? Hell yes I am tough enough."
"You need a toughness in this job. People have thrown a lot at me over four-and-half-years. I am a pretty resilient guy. And I have been underestimated at every turn. People said I wouldn't become leader and I did. People said four years ago 'he can't become prime minister', I think I can. You are saying I can't win a majority, I think I can. Let people underestimate me."
Pressed by Paxman on the idea that voters saw him as a "north London geek", Miliband replied: "Who cares?" The Labour leader said "the bloke on the tube can say what he likes".
He told the former Newsnight presenter and the studio audience: "Be yourself. That's what I am. People have to decide do they want my ideas, my principles. I don't care what the newspapers write about me. What I care about is what is happening to the British people.
As the interview came to a close. Paxman asked: "Are you ok Ed?" The Labour leader shot back: "Yeah, are you?"
Cameron edged the first major TV clash of the general election campaign against Miliband by a margin of 54% to 46%, according to a snap opinion poll released moments after the end of the 90-minute programme.
However the prime minister was also put through the mincer by Paxman, with the veteran presenter opening his line of questioning by raising his wealth. "What do you have in common with all these very rich people," he asked.
Nigel Farage, who had turned up the West London studio, said he thought Miliband had come out of the Paxman grilling more intact. "What was really interesting was the Prime Minister was deeply discomforted from the very start, didn't really get into his stride until about the 16th minute," he said.
"What surprised me about Ed Miliband, having said to you before the programme he might struggle against Paxman, is Paxman was just as tough on both of them on policy. But he asked Ed Miliband some very tough personal questions. Miliband sat there, took it, fought back, and actually showed some humour and got the audience on his side.
The Ukip leader added: "Taking policy out of it, on personality, for me, Miliband was ahead."