am i being silly or is my boyfriend selfish?

i want to know if I'm being silly or if my boyfriend is selfish. Basically both me and my children are very poorly my 11 month old is doing nothing but crying because he's not well and although my 3 year old is poorly she's still up to no good and sorting them out while ill my self is deffo a challenge. Anyway my partner works 9 till 10 and today he told me that he's going to the pub after work because he doesn't socialise enough (I never go out and he went out last weekend) I said to him that he isn't being fair because ive been looking after the kids while poorly myself and our son is very unsettled and won't sleep properly. I feel like a single mother because of his work hours as it is. i feel like my feelings don't matter when he's concerned and he never sees how he's upset me. No matter how many times I explain to him he just doesn't get it and carries on with doing things a single man would. usually he leaves me in the dark when he goes out and comes back at 3 in the morning drunk out of his face and I know tonight is going to be exactly the same.