Boris Johnson says his audiences are often "waiting like midwives" for him to give birth to a joke, when instead he has something "incredibly dull" to say.
But the colourful Mayor of London insisted he is too busy in his City Hall role to care about the way he is perceived.
Johnson has been tipped as a future Tory leader but critics say his flamboyant style prevents him being taken seriously.
Interviewed on BBC Radio 3's Night Waves about the way he is seen as a "performer", he said: "The risk is you just have something incredibly dull to say and people expect you to come up with a joke.
"You turn up at an event and I can see people hungering for the birth of the joke, they are waiting like midwives or staff at an operating theatre for me to produce this thing and nothing comes out and they look absolutely baffled.
"I simply say something completely reasonable. They want the gag and the gag doesn't come because there isn't a gag sometimes."
He added: "The answer to that is sometimes to have the wit to be dull... to dare to be dull."
Asked if he was worried he was seen as "not serious enough for the top job", Mr Johnson said: "I don't care about that. Being Mayor is unbelievably full of difficult executive decisions.
"I really don't have enough time to worry about that kind of thing.
"I've got almost three more years as Mayor... it's a long time, you can get a lot done in that period and I want to do that."
At last year's Tory Party conference, veteran Conservative minister Ken Clarke said Mr Johnson needed to "settle down" if he had ambitions beyond the mayoralty.
Mr Clarke said: "If he really wants to be a prime minister for serious reasons and not just getting his picture in the paper more often, he really does have to settle down and demonstrate he can seriously deliver on some complicated subjects."