We deal with this only it's not the adult children that invite her to everything because the children are young but its my in laws who invite her to everything because they don't want her feel offended that she isn't a part of the family anymore.
The fact of the matter is there is nothing YOU can do about it only YOUR HUSBAND can do something about it. Your husband has to tell the kids that they are invited and it is an invitation only kind of thing. It might make people uncomfortable at first but boundaries have to be established. Boundaries also need to be respected and the fact of the matter is, regardless of if BM is nice or not, she knows that she doesn't belong there but she is pushing the boundaries.
In my situation the BM goes to things because she knows we think its weird and she just does it because she can. She even shows up at family reunions sometimes. I've learned that there is nothing you can do about it. Maybe take the kids out individually. She won't come then.

Wow..just wow. I kind of understand your DH's dilemma. One...he has never said anything before so he does not want to start now. BUUUUT...he needs to put a stop to it anyway.

He needs to sit his kids down and tell them that next time he invites them all out for dinner he really wants it to just be them and the grans kids. Not their mother. He understands why they think they need their mother to come, but he really just wants to spend time with his kids and not his ex wife. Nothing against their mother, but he would rather spend time with them independent of her.

Would he be willing to do that or is he too scared. Are the kids that bad that they would actually get mad about this?

What gall. BM is way out of line and most likely knows it, but continues to pull this crap simply because she hasn't been called out on it. And she will continue to butt into your marriage and family until someone draws the line with her. I'd give my husband one chance to set her straight, but if he was unable to do it, for whatever reason (he need not fear the children will ditch him.. that's highly unlikely to happen as long as they think there's something in it for them), I'd absolutely get that mess sorted out myself. Your DH has no right whatsoever to even expect you to put up with that crap.

What they're doing is pure BS. And if it's not stopped NOW you'll be dealing with this BS forever. At some point, you're going to have to take a stand. Believe me.. taking that stand now is better than taking it after years of abuse from skids and BM. And this IS abuse. Call it for what it is. If it turns out DH doesn't have your back on this.. so much better to know this sooner rather than later. Forty nine years I played nice and put up with similar BS. NO MORE. Finally hit my quota lol