Baby no 2

I'm 26, mum of 4 year old boy. I lost my babies father when my son was 1,5. I have been in a relationship for almost 2 years. My partner is 2 years younger than me. Recently I have been trying to find out how he feels about having his own children as I'd like to have one more baby. My partner had a difficult past his mother and stepfather kicked him out of home when he was 16, his childhood wasnt very good and he doesn't keep in touch with any of his family. Since 16 he's been pretty much on his own. He never really had a proper relationship before me, nothing serious anyway and he doesn't socialise alot. At first it was difficult for him with my son but things are getting better. Him and I get on well and at first he kept saying maybe one day we can have one but recently in an argument he's told me were never having one and that if my son weren't already here he'd probably never have one at all. He wants a 'peaceful life'. He has explained that he's scared of the whole baby stage and finances and how it will put us behind. We are trying for a house and I've just gotten into a full time job. I have explained that I don't want one now, maybe in a year or two but this doesn't help... I'm afraid I'm rushing things but I just feel like the clock is ticking for me... I've been trying to accept the fact that maybe only 1 child is meant for me, but it's really hard. I grew up in a big family, with lots of cousins and a brother who I'm close with, I'd like my son to have a sibling. My partner keeps saying I don't compromise, he doesn't see anything wrong with this situation as I've already got 1 child so it's not like I'm childless, which I understand. Am I being too pushy?

Any advice would help
Thanks