should i stop my sons father having access to him?

My son is 3 years old and his dad is a part time father. He will go weeks without seeing him and does not contribute a penny towards his upbringing. I would have stopped contact months ago but my son worships the ground he walks on and is constantly asking to see him. I'm really stuck as to whether cut all ties and not let his father see him, but I can't help but feel guilty if I do. My son is beginning to realise that his daddy is not around and it is breaking my heart.