If he can that easily tell you to pack your stuff up and move out, do it. He's showing you right there the level of respect(or lack there of) that he has for you. 

You are not "thinking too much into it." You are only the bad guy because he is making you into the bad guy by telling the kids that them having to behave is because of you, instead of actually parenting. The fact that he said what he did indicates that he would rather give up than put in the work needed to resolve this pretty simple situation. This sounds cold, but either he cares very little for your relationship, or he is incapable of being a good partner. Either way, you deserve better. You are not asking too much. 

To start with, you should not be keeping an eye on them. They're not your kids, they are his responsibility. Second, if you're working, you can't keep an eye on them - dammit, woman, you're busy! Next off, he's a lazy apology for a father who doesn't know how to parent and doesn't care. To wind up, he's rude to you, makes you the bad guy (you're not) and thinks your disposable. So leave him to worry about his kids, the bills, and all the rest and pack your bags. Go make a good life elsewhere. It's the very least you deserve. Don't waste any more of your precious time and energy on such a lout.

Toss this one back, he's not a keeper. He expects you to watch his kids while you are working, won't discipline them appropriately and teach them to respect that you are working, and then tells you to pack your bags when you get into a disagreement.

If you are going to stay, stop watching his kids while you work, since neither he nor they respect your need for quiet.

Why can't you lock yourself up in a room away from them?  Why is that you have to work and watch them too but he doesn't have to watch them?  They HIS responsibility not yours.

I also hate to point out the obvious but he is making it sound that way when he tells the kids because that is exactly what he means.  Hey kids you could yell and scream and be as loud as you want if it wasn't for her.  It is her that wants you quiet not me.  His response by asking you to leave when you disagree is your window to see what your life will be like if you stay with this guy.  You are never going to be able to disagree or have an opinion about anything.  

If you can't talk this out I would get out now !!

He usually leaves me to keep an eye on them while he’s working so it’s not like I can just go to a different room and close the door all day— which is fine!

No, it's not fine, they are not your kids and not your responsibility. And your so called BF is not a keeper.

You've only given us a small glimpse of him but it's enough to know that he has no respect for you and takes you for granted.

You appreciate that he finally said something? Don't. He wasn't being respectful or teaching his kids to be respectful, he absolutely was being an ass to you.

He told me that “he can’t win with me” because I’m not happy when he says something and I’m not happy when he doesn’t.

Typical form of gaslighting, throwing it back at you, as if YOU are the bad guy, the problem, when it's actually him. And it's working because you are questioning yourself. Don't.

I would make a plan to do that packing up and moving out that he pretty much told you to do.

I'm sorry, but screw this guy! They aren't your kids! And even if you say you don't mind watching them, why is he the one who can work in peace and quiet while you do HIS job as a parent? And he is TOTALLY making you the bad guy! Like unless this guy has a golden dick and the stamina of an Olympic athelete, go find someone without kids (or even with kids but who's willing to be a good and active parent!) to be with! And heck, even if the D is golden, get a last good time or two in and then pack your bags. Especially since it's hard to find good D in the era of social distancing. But after that, RUN.

And yes to the comment above! You are doing him a freaking FAVOR and a BIG ONE watching the kids and then he acts like you're the problem??? NO. So much no. 

This disturbs me ---> "she will be mad if you’re making noise" because as already pointed out in the comments, he's100% throwing you under the bus so he can still look like the good guy. It's an "us vs them" stance and honey, you're "them". Not a comfortable place to be- I've been there with my fiance and his two boys. "Jeni doesn't like it when---" or "Jeni will be pissed when she finds out---" (not "we aren't ok with---") The alternative is he wants ME to tell them how I feel, which I do, but he won't back me up. Either way, it feels bad when the three of them have circled wagons and I'm on the outside. *awkward*

Have you two ever talked about values and boundaries when it comes to parenting? 

This dickhead has told you where you stand. Believe him and leave.

No man should speak to his mate that way. He has and in doing so has proven himself unworthy of you.

Find a man with character and who clearly comprehends that his mate and the relationship he shres with his mate are the unchallenged priority.  Kids certainly are and should be the top relationship responsibility but should never take priority over the mates or their relatiohship.

Take care of you.

Good luck.

Run now!