Wearing a poppy is not a comment on politics or military intervention. I doubt that everyone who wears a poppy agrees with all aspects of British foreign and military policy dating back to the first ever Poppy Day in 1921. If you object to British foreign policy, about the worst way you could express that is in a decision not to wear a poppy, because that decision only impacts on some of those who face consequences of the policy - whether or not they agree with it - not on those of us who are actually responsible for the decisions.
I should probably point out that I'm not against the original meaning behind Poppy Day: remembering how Britain twice sent a whole generation of its young men off to be slaughtered, and that future generations should be able to live without the fear of enduring such violence. Yet that meaning often seems to get forgotten as Remembrance Sunday becomes a celebration of jingoism and militarism, where the victims of British aggression in wars past and present are rarely mentioned.
There are not many things I dislike about living in London. Of course, the weather could be better sometimes; transport could definitely be improved upon; and an increase in the living wage would help most ordinary Londoners. But the one day of the year I have come to absolutely loathe and despise in over a decade of living in this great city is Remembrance Sunday.