I could have written your exact same story to 99.95% just a few months ago.  I finally found my worth and value and realized I wasn't going to put up with being in last place of my SO's life so I left.   I will tell you that it will not change...there is nothing that you will be able to do or say to get him to realize that he is the problem so you have a big decision to make.

Yes its very hard to leave and move on, but 1) you are not being valued as a partner now, 2) if his kids dont accept you now, they likely never will and do you really want family events like holidays and such to be that awkward forever? and 3) if you want a family and that is important to you then you need to find someone that can provide it to you.

I know its hard now, but just imagine what your life can look like with someone that loves you more than you can imagine, can give you the family that you want and that you can make new friends with.   Please trust me, I still do miss my ex and wish things could have been better but at some point you need to evaluate your life, your importance and what you want and expect in a relationship and move on if the one that you are currently with is not willing or is not capable of providing that to you.  Life is too short!

Welcome.

When you say he "left everything to be with you", does that mean you were his affair partner? If so, you can't expect this to settle down any time soon.

This is no life for you.  If you were this man's affair, and the reason he left BM, your life is going to be hell for ages.  Even if you weren't, he is tossing you a few crumbs, remaining enmeshed with his ex, and putting you a very poor second to both ex and kids.  Not even allowing you to discuss having your own child, which many women approaching 40 might want to do in pretty short order.   You should not have to put up with the SKIDs verbally and non verbally abusing you, when his back is turned.  You should tell him, and he should sort it out with his kids.  But he probably won't, and you should probably cut your losses and leave. 

How can you love someone who completely disregards your feelings? Who won't listen to your concerns? Who gaslights the hell out of you? Are you sure you love him or are you in love with the idea of what you wish he were? 

He's treating you as if you were a bit on the side who comes over to his place from time to time rather than a full-time partner. He's putting you after his ex-wife (!) and his kids. How does it feel to be at the bottom of the list? 

How do you picture your life ten years down the line? Still hiding away when the kids come over? Or worse, supposing he moves them in? Will you be resentful of not having a child of  your own? NOW is the time to decide. If he won't entertain your concerns then it's time you started planning for your future, whatever you may choose that to be.

You only live once and life is too short to throw it away on someone who likes having you around but who won't compromise and who imposes all his own choices while ignoring all of your requests... 

Two things jump out here.  Alluding to a possible affair and the possibility of a reconciliation.

This will not end well for you.  IF this man left his family to be with you,his kids will not respect you.

He may be with you simply because BM found out about the affair and tossed his butt out.  He  may interact with her as much as he does because he is hoping  she will take him back. 

 You say that you  know that BM will take him back.  So it seems that they both may want to work on their issues and get back together.

I'll suggest that you find a counsellor and try to understand why you are so in love with someone who has little to no consideration for you.  You want kids, he's told you he doesn't want anymore. You wrote, " I can't really talk to him about any of the big stuff...kids, BM, my feelings etc, he just either gets upset or angry and defensive."  That is not how a man who loves and cherishes you reacts to important conversations.  You deserve better!  

This is what's called "having ones cake and eating it, too". He hasn't really given up everything to be with you, now has he? You are living where HE wants to live, doing what HE wants to do. He is still enmeshed with BM, his kids still come over everyday, and he lets them disrespect you. He calls all of the shots.

You are basically still the mistress.

This will not end well - it was dead in the water before it even began. When you begin a relationship with someone dishonestly (no judgment thats not the point here...) it is flawed from its inception. It is begun on shaky ground. It is weak, not strong, built on lies and deception. Affairs that end up in relationships, when there are children involved especially, cannot be made clean and whole again. Love is not enough.

As other have said and you made comments of, you are still his "side-peice" and his main wife is still the ex, the mother of his 3 children. The children will never forgive you. Ex is not mad, because she still has "her man". She is definitely disordered. He is MAJORLY disordered. Forget about not being his priority, forget about him putting you and your life together last, he is too disordered, this whole relationship is too disordered to be at all healthy for you.

Get your ducks in a row financially. Let the ex have her man back, get out before you waste anymore time with this narcissistic POS. He betrayed his family and you will ALWAYS be last. He is not worth it. Time to move out and move onward.

Other than possible feelings of guilt because this man left his marriage for you. What are you getting out of this relationship? You are insecure,  with good reason. You dont like his children,  and they will probably never like you. You want a child of your own, which he is not going to give you. If you rock the boat too har and he doesnt want to deal with it, he has the option to go back to his wife. 

The kids will NEVER like you or respect you.  NEVER.  

To address your DHs demeanor.  Despite what you have heard, read or been told...(even if by BM) I would bet money that DH actually asked BM to stay with him and she said NO.  I bet he has asked her to get back together many times over the last year and she said no.  JMHO.  

Find a single man.  

I was in your situation 8 years ago- and some of the things you wrote we're the same or similar to what I dealt with. I was 26 and he was 38- 3 kids 10 (g),12 (b),15(b). There is so much judgment that it's hard to admit when you got together due to an affair. But each situation is very unique and sometimes you just fall in love with someone you shouldn't- I met my DH by complete chance, not sure on your scenario.

BM made sure the kids knew all the details about what their DH did and really screw them up for life. The boys were pretty cordial with me from the beginning because they weren't close with their Mother (a lot of issues with her growing up) and soon enough his Daughter would see the same things they did. If the marriage didn't work out it wasn't JUST because of your affair. I believe there has to be many other things wrong to lead to the affair and especially to go through with a divorce- esp since she didn't want it.

I don't like that he's talking to her multiple times a day and when he's not around you. I told DH that was sketchy when he did that and that anything they were discussing should be able to be discussed with me around. Not that I needed to be in the same room. I'm sure he thinks he's doing it not to upset you but it's only making the suspicions of what they're talking about grow.

I used to check the phone as well- STOP. You will drive yourself insane. I did that for years in the beginning. I do believe a big negative in starting a relationship having an affair is that there will always be some trust issues. My DH's BM would intentionally send texts that would piss me off... being all sweet and friendly. She also wanted him back- or at least wanted to mess with me. Can't blame her- she was hell bent on revenge. But you have to make the decision to trust him or leave him.

I have no kids of my own either. You are probably always going to feel that resentment or longing to have that with him especially since his own kids are awful to you. That will just make you want that more. It's a HUGE thing to give up. I never liked or wanted kids- I was OK since his were older. I only ever questioned wanting a child until I felt the jealousy of him having that with his ex and feeling like an outsider.

My DH worked really hard to encourage a relationship with his daughter and she and I eventually got better (despite my recent blogs- LOL). We were really close for awhile and believe me- this kid HATED me. But we did some fun family outing type stuff and eventually she gave in. Now she's just an annoying teenage girl with a chip on her shoulder- but our relationship is OK. We are not close but part of that is because I have chosen to disengage from her after some things she did to really upset me. But it's manageable. The boys are 19 and 23- we get along very well now. The oldest gives hugs and says he loves me.

I share all this with you because this started 8 years ago for me. I thought my situation was HOPELESS. I thought those kids would NEVER accept me. But 4 years ago we moved in together and 2 years ago we got married. Things are much better all around. BM doesn't bother him anymore (knock on wood) and I finally feel like we have our own life.

Maybe you could too... but DH is going to have to stand up to these kids and minimally they need to be respectful while they are at your home. I don't envy the place you are in. If I had to do it all over again- UGH- I would but good God I wouldn't want to. LOL. Make sure he's worth it.. it's not going to be an easy road if you stay.
HUGS~~~

You say that you are afraid of losing him. Ask yourself WHY that is.

How long did you date after his divorce? Was it enough time to get to know him as a real, whole person? Living as a secret girlfriend is no fun in many ways, but when you were with him then there wasn't the pressure of normal life such as kids, exwife, household bills and laundry. There is a reason affair partners rarely make the successful transition to married partners. Not saying never, but it's rare.

Find a good therapist to figure out what what's going on with you. Why the affair in the first place (and I'm not judging)? Why hang on to a relationshp that you're not really an equal part of, with a guy you can't even talk to?

Regardless of whether being his affair partner was right or wrong, you are paying for it and will continue to pay if you stay with him. The fact that he "chose you" will be held over your head and used to justify all kinds of bad behavior from DH, BM, and skids. It already is. Nobody should put up with what you do. 
 

IMO your best bet is to take this experience as a lesson. Value yourself and don't accept scraps from an unavailable man or disrespect from kids in the future.