Back story (I'm not familiar with the abbreviations you guys use yet, so I'm just going to try to keep it brief instead)

My guy SO and I have been together almost 2 years. He has two ex wives and one child with each. Daughter is 21 and lives with mom and is a moderate visitor with her dad. Son is 11 and is in shared custody with SO and his mom. I have 4 kids, three of them still live at home, their ages are 26, 17, 16, and 13. Two weeks into our relationship, SS mom tells him that I am a pole dancer and that she doesn't think that it's good for him to be around me. I almost walked out then. But SO told me he really liked me and didn't want her to win so he yelled at her and that was that. That was February. In july, he ended our relationship because he was finalizing his divorce with SS mom and she was basically demanding everything and he was terrified of what the financial outcome would be if he didn't cave so the easiest route was for him to end our relationship. Except what he told me and the ex, and his kids, is that I would not allow him to coparent effectively because I wanted boundaries. 3 weeks later he wanted to get back together, and I agreed after telling him that he should not have lied about why he ended it. Then in November he broke it off again because he wasn't sure he wanted to get married like I do, and told his daughter that we won't be getting back together but then changed his mind again and we got back together after I told him he has to be authentic and it's okay to not want what I want as long he's honest. After this, his daughter started acting very avoidant and made him promise to not talk about me and not involve her in our relationship. I wasn't aware until recently that she felt so strongly about me, or our relationship. So I was trying to be friendly with her, liking her social media pictures, occassional supportive messages, what have you. Trying to be friends. Not aware that she felt the way she did. But eventually picked up on that notion and realized she didn't want me around her. I started avoiding acitivities for his son, her brother,  just to give her the space from me. Well SO kept asking me to go and  I refused thinking that was the right thing to do.Until one day a few weeks ago he messaged her and said they need to have a talk and get to the bottom of whats going on and how to make amends. She flipped out and said it's no longer about me, it's about him and him breaking his promise and she doesn't want to be involved period and he better stop asking her. So that caused some tension. In June we all moved in together, before I knew she felt so strongly. Things were going well with his son and my kids. But the tension of me not going and his daughter flipping out took it's toll and we got into a fight a couple weeks ago. SO stormed into the house and told his son to get his stuff so he could get dropped off at his moms. I remained calm and told him it was okay and that SS should finish his dinner first and said I needed to run an errand. Just to diffuse the situation. Well after I left , he told his son that they were moving out! In front of my kids. So my daughter told him that was so unfair to say and that he was wrong. Later that night SO decided he was wrong for saying that to his son and in front of my kids, but now SS is uncomfortable. He told his mom, who told his sister and her mom, and now they are all against me/us. Last weekend was the last baseball tournament for SS and they all went and had a great time while I was excluded. Needless to say that hurt like hell and I have done nothing but try to fit in and be nice. His exes and kids can't say a bad word about me, so do you know what they say about me? They say I'm 'too much' and 'try too hard' because i have given gifts or reached out to be friendly and welcoming versus looking like I'm trying to steal my SO from them all. Its incredulous to me that they can use niceness against me! And now SO feels like an ass for badmouthing me and leaving me over and over, but the damage has been done. I have insisted we start counseling to learn some tools to help us and deal with this mess. But I'm feeling pretty discouraged and like I'll always be an outsider and I'm not sure I'm strong enough for that. But I do love him and want so badly for it to work out. Now, on August 14 there is a baseball game that SO wants me to attend with him, while his daugter goes with her brother and his mom as a group and we are on our own. He will go no matter what to support his son, as he should, but I'm not sure my going is a good idea? He says we have to show them we are a united front and that I'm not going anywhere. But I feel like I'll just be crying on the inside the whole time knowing they all hate me and don't want me there. Is this how it's always going to be? Always having to be on the outside or feel like trash when I'm around them? I'm rambling I know, but I have no flipping idea what to do here and some sage advice would be really awesome right now. Help!