My DH took BM back to court in March 2014. There were several appearances before the actual trial. The last appearance (in October before the Mar trial) the judge told her to cut the harassment. THE VERY NEXT DAY she amps it up to an UNBELIEVABLE new level.
After court in March, and DH won, it has never been better. But to get to this point was nightmarish.

"Bm is very fearful of the law and is very concerned about what people think about her as a mother"

Well, the same can be said about my DH's ex but things are still really bad despite the existence of a CO. She can find ways to justify anything in her own mind. This woman can tell herself the sky is green and begin to believe it after a while. Everyone is prone to their own cognitive biases but she is the queen of it.

I hope your BM is smarter than ours.

In our case with BM1 she finally backed off.

With BM2 on the other hand, she would always try her hardest to be super nice, which means for normal people not yelling and screaming. But it never lasted.

The last time we went to court, she has calmed down a lot. We hardly ever hear from her and it has been very peaceful.

DH served BM with papers in Oct 2013, we had a couple of court dates and the first trial date was set for April 2014. After the Jan 1, 2014 BM truly went Bat Shit Crazy and got her first arrest, then in March 2014 she "attempted" suicide (attention was needed) and CPS in the state she was living in took the kids and we gained custody. Sept 2014 we finally got our trial in our state where the divorce took place and BM signed over sole custody legal and physical to DH.

But before she completely lost it she was trying to be nicer and actually do right for the kids, but I think she was only being that way because she knew we were watching her every move.

For me yes.
Long story short BM and my partner were never married, BM cheated when SD was newborn and she then left SD with my partner. My partner was FT CP from baby till SD was 2. When BM found out we were dating, literally very very early days she seemed to go a bit psycho paranoid about someone else wanting to swoop in and play mum to SD. She took my partner back to court and gained custody based on "mothers lost time" and because my partners then working hours werent as convenient as hers.
In the very early days she did seem a bit odd, she rang my partner one day after first seeing me demanding to know things about me, my partner shut her down and said he doesnt ask her anything about her personal life she has no reason to know his.

As soon as the CO was in place BM was happy, she is also someone who cares what people think and for her it was really important that she was the CP for SD starting (pre)school. She sticks to the CO, my partner sticks to the CO there is very rarely contact between them as most handovers take place via school. Contact is civil and we have had zero drama.
SD is 4 now so Im two years in, been living together just shy of 18 months.

It can happen that once a CO is in place and everyone knows where things stand that there it can settle but it always depends on the motivation of the BM. Our BMs motivation was gaining custody of her daughter, shes got it so shes happy. BMs whose motivation is to cause heartache to their exes will never settle regardless of COs, ROs or whatever.

"I get that there doesn't need to be a relationship between Bm and I but it would be nice for her not to have so much hostility toward me because it's not healthy for Ss"

I haven't had the pleasure of going to court as I met DH post divorce and there haven't been any updates to the CO in the last 7 years.
However, I wouldn't hold my breath. In my 6+ years of experience with the crazy BM in our situation, the stepparent is always the scapegoat for anything and everything. Hostility ramps up anytime the BM has a feeling of any kind.
Some examples:
BM's relationship doesn't work out; angry at SM because DH isn't available to go back to, drama ensues.
Kid likes something SM did or said or gave kid; jealous of SM because she's making BM look bad, drama ensues.
DH or SM get something new (house, car, job, hell even outfit); jealous so try to get more money, drama ensues
BM has a birthday; jealous because SM is younger than she is, drama ensues.
DH does anything that shows he loves SM, like get a sitter for Valentines Day, jealous because no one took her out, drama.
Etc, etc, etc.
This happens even on occasions that BM is genuinely trying to improve things. As soon as her feelings are hurt, BOOM, drama.

As for the sake of the kids, if BM was capable of putting their needs before her own, there probably wouldn't have been a divorce in the first place (assuming your DH is a decent guy).

BM tried to take my husband to court about 6 months ago just to have the CO changed, which they've always done through their lawyers in the past rather than going to court. I think she just wanted to feel powerful.

She had to pay a $500 fee to file with the court and she tried to get my husband to pay for it. His lawyer told her lawyer that it wasn't a litigious matter so there was no point in filing a court date to begin with, so my husband didn't have to pay.

We're going to be moving to a new town soon and my husband will need their meeting place to change. He's planning on just asking BM about it so they can work it out themselves, but I have a feeling she won't agree because she'll have to drive a little farther. If she doesn't agree, he'll have to go through his lawyer to get it done. I don't think she'll want to pay her lawyer more fees after the negotiations they just went through this summer, so hopefully she'll cave.

I agree with people above who have said that if you win a court case against BM, she's probably less likely to try something again in the future.

Going to court was a good thing for us in that it gave us a tool to decrease the toxicity in the kids lives. Your best bet is to get the court order written in a way that spells everything out. If BM is fearful of the law, this might get her to toe the line. Many judges are pro-BM, but don't like it when someone breaks their orders. This was the case with the judge DH and BMs attorney went before.

Document everything. I can't emphasize that enough.

It's a yes and a no. Bm was horrified that she actually lost. She went in thinking because she was Mom she would get week days and every other weekend and child support. She gets no child support and gets SS on the weekends until Summer then it's 50/50 like we had done previously (something she was begging the judge to go back to once he ruled in favor of DH). After court she was pissed. She would cause problems anywhere the CO wasn't specific or clear. She went back got the transcripts and tries to find something me or DH lied about. She constantly threatens to go back. But generally she will pick up and drop (as long as it's clearly written where and when). But she didn't return SS on a holiday that was suppose to be DHs and we called the cops. They did NOTHING. Now that she knows CO are basically useless pieces of paper, she follows it until she wants something then she just does what she wants because the cops won't enforce anything but trespassing.

In her eyes, our BM lost when we last went to court. DH felt like he lost. She has never gotten over it. It has gotten increasingly worse with her ever since. DH has paid thousands this year alone to fight certain issues she has taken to a lawyer. The most recent one is currently with the judge to decide. I fear it will get even worse if the judge rules against her this time.