Very difficult baby - how do I go out and have a life?

My little one is 8 weeks old, he has quite severe colic, reflux and cows milk protein allergy. He is pretty miserable most of the time, cries constantly for most of his awake time. We have started him on new prescribed formula and I’m starting to see a difference but he’s still very uncomfortable and grumpy. 

I’m really struggling to deal with it (translation - I’m losing my mind haha) and I really want to start getting out of the house and going to baby groups but he’s a nightmare to take out. He doesn’t sleep well unless swaddled, in his cot, white noise and blackout blinds. Every time we do go out it’s awful and he cries for pretty much the whole time and then I don’t enjoy myself at all.

He’s in a pretty good routine and I’m scared to upset that by taking him out and then him not sleeping and then getting overtired. He feeds around 7am, 10am, 1pm, 4pm, 7pm and naps for between 45 minutes - 2 hours in between feeds. He can only manage 1 hour - 1 hour 15 awake time and most of that is taken up by feeding, winding and nappy changes. 

Does anyone have any tips for how to get a baby used to going out and about? Anyone in a similar place or been through it?