Baby’s Surname - ex-partner wants his name for baby

hello all, wanted to ask on here to get a fair non biased opinion but my son Is now 5 months and has my surname from birth, the father wasn’t around for the first 2 months but then he became interested and see’s him once a week. He is now asking to double barrel babies surname but I’m not to sure. Obviously all my family agree with me and his family agree with him so I was just wondering what everyone else’s thoughts are on this? Am I being unfair not wanting to change it?