Although it is a well known fact, there are still some people who are totally unaware that the people with whom they have the most contact and hang around will have either a positive or negative effect on their levels of self-esteem and confidence.
A snide comment, a patronising remark, a withering look: it sounds like the childish behaviour of kids in the playground, but it happens daily between adults in the workplace too, and can have damaging consequences.
If you are working ridiculous hours to get things done, unfortunately, you only have yourself to blame. Go home when you are supposed to every night this week and see what happens. You might get a talking to by your boss, but I can guarantee that you won't lose your job.